guides/developer/RPC-API.md
0 → 100644
Version 0.4.1 | ||
------------- | ||
First of all, you should take a look at the | ||
<a href="https://github.com/Arachni/arachni-rpc">Arachni-RPC</a> | ||
protocol itself especially its | ||
<a href="https://github.com/Arachni/arachni-rpc/wiki">design specification</a>. | ||
To provide functional examples of RPC interaction, I"ll be using the | ||
<a href="https://github.com/Arachni/arachni-rpc-pure">pure Ruby client | ||
implementation</a> of the Arachni-RPC protocol. | ||
If you are interested in the full API you can browse through it | ||
[here](http://arachni.github.com/arachni/Arachni/RPC/Server.html) -- the | ||
accessible classes are Dispatcher, Instance and Framework. | ||
In order to perform an audit you have to go through the following | ||
straightforward steps: | ||
1. [Connect to an Arachni RPC Dispatch server](#dispatcher_connect) | ||
2. [Request an Arachni instance](#dispatch) | ||
3. [Retrieve framework components (Optional)](#retrieve) | ||
1. [Modules](#retrieve_modules) | ||
2. [Plugins](#retrieve_plugins) | ||
4. [Set the options you want](#options) | ||
1. [Setting the URL (opts.url=)](#options_url) | ||
2. [Audit links (opts.audit_links=)](#options_audit_links) | ||
3. [Audit forms (opts.audit_forms=)](#options_audit_forms) | ||
4. [Audit cookies (opts.audit_cookies=)](#options_audit_cookies) | ||
5. [Audit headers (opts.audit_headers=)](#options_audit_headers) | ||
6. [Link count limit (opts.link-count-limit=)](#options_link_count_limit) | ||
7. [Exclude cookies (opts.exclude_cookies=)](#options_exclude_cookies) | ||
8. [Exclude vectors (opts.exclude_vectors=)](#options_exclude_vectors) | ||
9. [User agent (opts.user_agent=)](#options_user_agent) | ||
10. [Exclude patterns (opts.exclude=)](#options_exclude) | ||
11. [Include patterns (opts.include=)](#options_include) | ||
12. [Cookie jar (opts.cookie_jar=)](#options_cookie_jar) | ||
13. [HTTP request limit (opts.http-req-limit=)](#options_http_req_limit) | ||
14. [Redundancy rules (opts.redundant=)](#options_redundant) | ||
5. [Load the modules you want to run](#modules) | ||
6. [Load the plugins you want to run](#plugins) | ||
7. [Run the framework](#run) | ||
1. [Get the progress while busy](#run_busy) | ||
2. [Pausing and Resuming](#run_pause_resume) | ||
8. [Get the report](#report) | ||
9. [Shutdown the server](#shutdown) | ||
10. [Cancelling the scan](#cancel) | ||
### <a id="example" href="#example">A minimalistic example</a> | ||
Setting up the test environment | ||
------------------------------- | ||
First of all, we"ll need to run an Arachni RPC Dispatcher to have | ||
something to work and play with. | ||
$ arachni_rpcd | ||
Arachni - Web Application Security Scanner Framework v0.4.1dev | ||
Author: Tasos "Zapotek" Laskos <[email protected]> | ||
(With the support of the community and the Arachni Team.) | ||
Website: http://arachni-scanner.com | ||
Documentation: http://arachni-scanner.com/wiki | ||
Arachni - Web Application Security Scanner Framework v0.4.1dev | ||
Author: Tasos "Zapotek" Laskos <[email protected]> | ||
(With the support of the community and the Arachni Team.) | ||
Website: http://arachni-scanner.com | ||
Documentation: http://arachni-scanner.com/wiki | ||
I, [2012-08-28T05:29:39.412457 #23997] INFO -- System: RPC Server started. | ||
I, [2012-08-28T05:29:39.412557 #23997] INFO -- System: Listening on localhost:1605 | ||
Arachni - Web Application Security Scanner Framework v0.4.1dev | ||
Author: Tasos "Zapotek" Laskos <[email protected]> | ||
(With the support of the community and the Arachni Team.) | ||
Website: http://arachni-scanner.com | ||
Documentation: http://arachni-scanner.com/wiki | ||
I, [2012-08-28T05:29:39.419214 #24001] INFO -- System: RPC Server started. | ||
I, [2012-08-28T05:29:39.419314 #24001] INFO -- System: Listening on localhost:54160 | ||
[...lots of similar output...] | ||
This is what happens when no options have been set; the default port is _7331_. | ||
<a id="dispatcher_connect" href="#dispatcher_connect">Connect to an Arachni RPC Dispatch server</a> | ||
--------------------------------------------------------------- | ||
First of all, install the Arachni-RPC Pure client: | ||
gem install arachni-rpc-pure | ||
Simple as: | ||
```ruby | ||
require 'arachni/rpc/pure' | ||
dispatcher = Arachni::RPC::Pure::Client.new( | ||
host: 'localhost', | ||
port: 7331 | ||
) | ||
``` | ||
<a id="dispatch" href="#dispatch">Request an Arachni instance</a> | ||
---------------------------------------- | ||
```ruby | ||
# request for an instance to be dispatched | ||
instance_info = dispatcher.call( 'dispatcher.dispatch' ) | ||
``` | ||
<a id="instance_connect" href="#instance_connect">Connect to the Arachni RPC instance</a> | ||
-------------------------------------------------------- | ||
Simple as: | ||
```ruby | ||
host, port = instance_info['url'].split( ':' ) | ||
instance = Arachni::RPC::Pure::Client.new( | ||
host: host, | ||
port: port, | ||
token: instance_info['token'] | ||
) | ||
``` | ||
**In order to successfully authenticate yourself to the instance don't forget | ||
to include the authentication token.** | ||
<a id="retrieve" href="#retrieve">Retrieve framework components (Optional)</a> | ||
----------------------------------------------------- | ||
This is strictly optional and is only useful when you"re developing an interface | ||
and want to show the user all available components -- or in similar situations. | ||
### <a id="retrieve_modules" href="#retrieve_modules">Modules (framework.lsmod)</a> | ||
To retrieve all available modules: | ||
```ruby | ||
instance.call( 'framework.lsmod' ) | ||
``` | ||
Which will return something like: | ||
[ | ||
[ 0] { | ||
:name => "Code injection", | ||
:description => "It tries to inject code snippets into the\n web application and assess whether or not the injection\n was successful.", | ||
:elements => [ | ||
[0] "form", | ||
[1] "link", | ||
[2] "cookie", | ||
[3] "header" | ||
], | ||
:author => [ | ||
[0] "Tasos \"Zapotek\" Laskos <[email protected]>" | ||
], | ||
:version => "0.1.6", | ||
:references => { | ||
"PHP" => "http://php.net/manual/en/function.eval.php", | ||
"Perl" => "http://perldoc.perl.org/functions/eval.html", | ||
"Python" => "http://docs.python.org/py3k/library/functions.html#eval", | ||
"ASP" => "http://www.aspdev.org/asp/asp-eval-execute/", | ||
"Ruby" => "http://en.wikipedia.org/wiki/Eval#Ruby" | ||
}, | ||
:targets => [ | ||
[0] "PHP", | ||
[1] "Perl", | ||
[2] "Python", | ||
[3] "ASP", | ||
[4] "Ruby" | ||
], | ||
:issue => { | ||
:name => "Code injection", | ||
:description => "Arbitrary code can be injected into the web application\n which is then executed as part of the system.", | ||
:tags => [ | ||
[0] "code", | ||
[1] "injection", | ||
[2] "regexp" | ||
], | ||
:cwe => "94", | ||
:severity => "High", | ||
:cvssv2 => "7.5", | ||
:remedy_guidance => "User inputs must be validated and filtered\n before being evaluated as executable code.\n Better yet, the web application should stop evaluating user\n inputs as any part of dynamic code altogether.", | ||
:remedy_code => "", | ||
:metasploitable => "unix/webapp/arachni_php_eval" | ||
}, | ||
:mod_name => "code_injection", | ||
:path => "/home/zapotek/workspace/arachni/modules/audit/code_injection.rb" | ||
}, | ||
[ 1] { | ||
:name => "PathTraversal", | ||
:description => "It injects paths of common files (/etc/passwd and boot.ini)\n and evaluates the existence of a path traversal vulnerability\n based on the presence of relevant content in the HTML responses.", | ||
:elements => [ | ||
[0] "form", | ||
[1] "link", | ||
[2] "cookie", | ||
[3] "header" | ||
], | ||
:author => [ | ||
[0] "Tasos \"Zapotek\" Laskos <[email protected]>" | ||
], | ||
:version => "0.2.6", | ||
:references => { | ||
"OWASP" => "http://www.owasp.org/index.php/Path_Traversal", | ||
"WASC" => "http://projects.webappsec.org/Path-Traversal" | ||
}, | ||
:targets => [ | ||
[0] "Unix", | ||
[1] "Windows", | ||
[2] "Tomcat" | ||
], | ||
:issue => { | ||
:name => "Path Traversal", | ||
:description => "The web application enforces improper limitation\n of a pathname to a restricted directory.", | ||
:tags => [ | ||
[0] "path", | ||
[1] "traversal", | ||
[2] "injection", | ||
[3] "regexp" | ||
], | ||
:cwe => "22", | ||
:severity => "Medium", | ||
:cvssv2 => "4.3", | ||
:remedy_guidance => "User inputs must be validated and filtered\n before being used as a part of a filesystem path.", | ||
:remedy_code => "", | ||
:metasploitable => "unix/webapp/arachni_path_traversal" | ||
}, | ||
:mod_name => "path_traversal", | ||
:path => "/home/zapotek/workspace/arachni/modules/audit/path_traversal.rb" | ||
}, | ||
[...and many more...] | ||
] | ||
### <a id="retrieve_plugins" href="#retrieve_plugins">Plugins (framework.lsplug)</a> | ||
To retrieve all available plugins | ||
```ruby | ||
instance.call( 'framework.lsplug' ) | ||
``` | ||
[ | ||
[ 0] { | ||
:name => "Resolver", | ||
:description => "Resolves vulnerable hostnames to IP addresses.", | ||
:author => [ | ||
[0] "Tasos \"Zapotek\" Laskos <[email protected]>" | ||
], | ||
:tags => [ | ||
[0] "ip address", | ||
[1] "hostname" | ||
], | ||
:version => "0.1.1", | ||
:plug_name => "resolver", | ||
:path => "/home/zapotek/workspace/arachni/plugins/defaults/resolver.rb", | ||
:options => [] | ||
}, | ||
[ 1] { | ||
:name => "Health map", | ||
:description => "Generates a simple list of safe/unsafe URLs.", | ||
:author => [ | ||
[0] "Tasos \"Zapotek\" Laskos <[email protected]>" | ||
], | ||
:version => "0.1.3", | ||
:plug_name => "healthmap", | ||
:path => "/home/zapotek/workspace/arachni/plugins/defaults/healthmap.rb", | ||
:options => [] | ||
}, | ||
[...and many more...] | ||
] | ||
<a id="options" href="#options">Set the options you want</a> | ||
------------------------------------ | ||
You can do that in 2 ways, either set them all at once or one at a time. | ||
To set them all at once: | ||
```ruby | ||
# you can also use Strings as keys | ||
opts = { | ||
url: 'http://demo.testfire.net', | ||
audit_links: true, | ||
audit_forms: true, | ||
audit_cookies: true, | ||
audit_headers: true, | ||
link_count_limit: 1, | ||
user_agent: "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-us) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3Safari/5", | ||
} | ||
p instance.call( 'opts.set', opts ) | ||
#=> true | ||
``` | ||
You can use any of the available options that will be presented later in the ```opts``` hash. | ||
### <a id="options_url" href="#options_url">Setting the URL (opts.url=)</a> | ||
**Expects**: String <br/> | ||
**Default**: <n/a> | ||
To set the URL of the site you want to audit: | ||
```ruby | ||
p instance.call( 'opts.url=', 'http://demo.testfire.net' ) | ||
#=> "http://demo.testfire.net" | ||
``` | ||
### <a id="options_audit_links" href="#options_audit_links">Audit links (opts.audit_links=)</a> | ||
**Expects**: Boolean <br/> | ||
**Default**: false | ||
To audit link elements: | ||
```ruby | ||
p instance.call( 'opts.audit_links=', true ) | ||
#=> true | ||
p instance.call( 'opts.audit', :links ) | ||
#=> true | ||
p instance.call( 'opts.audit?', :links ) | ||
#=> true | ||
p instance.call( 'opts.dont_audit', :links ) | ||
#=> true | ||
p instance.call( 'opts.audit?', :links ) | ||
#=> false | ||
``` | ||
### <a id="options_audit_forms" href="#options_audit_forms">Audit forms (opts.audit_forms=)</a> | ||
**Expects**: Boolean <br/> | ||
**Default**: false | ||
To audit form elements: | ||
```ruby | ||
p instance.call( 'opts.audit_forms=', true ) | ||
#=> true | ||
p instance.call( 'opts.audit', :forms ) | ||
#=> true | ||
p instance.call( 'opts.audit?', :forms ) | ||
#=> true | ||
p instance.call( 'opts.dont_audit', :forms ) | ||
#=> true | ||
p instance.call( 'opts.audit?', :forms ) | ||
#=> false | ||
``` | ||
### <a id="options_audit_cookies" href="#options_audit_cookies">Audit cookies (opts.audit_cookies=)</a> | ||
**Expects**: Boolean <br/> | ||
**Default**: false | ||
To audit cookies: | ||
```ruby | ||
p instance.call( 'opts.audit_cookies=', true ) | ||
#=> true | ||
p instance.call( 'opts.audit', :cookies ) | ||
#=> true | ||
p instance.call( 'opts.audit?', :cookies ) | ||
#=> true | ||
p instance.call( 'opts.dont_audit', :cookies ) | ||
#=> true | ||
p instance.call( 'opts.audit?', :cookies ) | ||
#=> false | ||
``` | ||
### <a id="options_audit_headers" href="#options_audit_headers">Audit headers (opts.audit_headers=)</a> | ||
**Expects**: Boolean <br/> | ||
**Default**: false | ||
To audit headers: | ||
```ruby | ||
p instance.call( 'opts.audit_headers=', true ) | ||
#=> true | ||
p instance.call( 'opts.audit', :headers ) | ||
#=> true | ||
p instance.call( 'opts.audit?', :headers ) | ||
#=> true | ||
p instance.call( 'opts.dont_audit', :headers ) | ||
#=> true | ||
p instance.call( 'opts.audit?', :headers ) | ||
#=> false | ||
``` | ||
### <a id="options_link_count_limit" href="#options_link_count_limit">Link count limit (opts.link_count_limit=)</a> | ||
**Expects**: Integer <br/> | ||
**Default**: infinite | ||
To limit how many pages will be crawled and audited: | ||
```ruby | ||
p instance.call( 'opts.link_count_limit=', 1 ) | ||
#=> 1 | ||
``` | ||
## #<a id="options_exclude_cookies" href="#options_exclude_cookies">Exclude cookies (opts.exclude_cookies=)</a> | ||
**Expects**: Array of Strings <br/> | ||
**Default**: [] | ||
To exclude cookies from the audit process: | ||
```ruby | ||
p instance.call( 'opts.exclude_cookies=', [ 'sessionid', 'some_auth_cookie' ] ) | ||
#=> ["sessionid", "some_auth_cookie"] | ||
``` | ||
### [Exclude vectors (opts.exclude_vectors=)](#options_exclude_vectors) | ||
**Expects**: Array of Strings <br/> | ||
**Default**: [] | ||
To exclude input vectors from the audit process: | ||
```ruby | ||
p instance.call( 'opts.exclude_vectors=', [ 'password', 'csrf_token' ] ) | ||
#=> ["password", "csrf_token"] | ||
``` | ||
### [User agent (opts.user_agent=)](#options_user_agent) | ||
**Expects**: String <br/> | ||
**Default**: Arachni/\<version\> | ||
To set the user agent: | ||
```ruby | ||
p instance.call( 'opts.user_agent=', 'FooBar/0.1' ) | ||
#=> "FooBar/0.1" | ||
``` | ||
### <a id="options_exclude" href="#options_exclude">Exclude patterns (opts.exclude=)</a> | ||
**Expects**: Array of patterns (String or Regexp) <br/> | ||
**Default**: [] | ||
To set an exclude rule: | ||
```ruby | ||
p instance.call( "opts.exclude=", ["do_not_follow"] ) | ||
#=> [/do_not_follow/] | ||
``` | ||
URLs that match any of the patterns in the array will be ignored. | ||
### <a id="options_include" href="#options_include">Include patterns (opts.include=)</a> | ||
**Expects**: Array of patterns (String or Regexp) <br/> | ||
**Default**: [] | ||
To set an include rule: | ||
```ruby | ||
p instance.call( "opts.include=", ["only_follow_me"] ) | ||
#=> [/only_follow_me/] | ||
``` | ||
Only URLs that match any of the patterns in the array will be followed and audited. | ||
### <a id="options_cookie_jar" href="#options_cookie_jar">Cookie jar (opts.cookie_jar=)</a> | ||
**Expects**: Hash <br/> | ||
**Default**: {} | ||
To set the cookie jar: | ||
```ruby | ||
cookies = { | ||
"userid" => 1, | ||
"sessionid" => "fdfdfDDfsdfszdf" | ||
} | ||
p instance.call( "opts.cookie_jar=", cookies ) | ||
#=> {"userid"=>1, "sessionid"=>"fdfdfDDfsdfszdf"} | ||
``` | ||
### <a id="options_http_req_limit" href="#options_http_req_limit">HTTP request limit (opts.http_req_limit=)</a> | ||
**Expects**: Integer <br/> | ||
**Default**: 20 | ||
To limit how many concurrent HTTP request are sent: | ||
```ruby | ||
p instance.call( "opts.http_req_limit=", 20 ) | ||
#=> 20 | ||
``` | ||
**Note**: If your scan seems unresponsive try lowering the limit. <br> | ||
**Warning**: Given enough bandwidth and a high limit it could cause a DoS. Be | ||
careful when setting this option too high, don"t kill your server. | ||
### <a id="options_redundant" href="#options_redundant">Redundancy patterns (opts.redundant=)</a> | ||
**Expects**: Array of Hashes <br/> | ||
**Default**: [] | ||
```ruby | ||
redundant = { "follow_me_3_times" => 3, /follow_me_5_times/ => 5 } | ||
p instance.call( "opts.redundant=", redundant ) | ||
#=> {/follow_me_3_times/=>3, /follow_me_5_times/=>5} | ||
``` | ||
<a id="plugins" href="#plugins">Load the plugins you want to run</a> | ||
-------------------------------------------- | ||
**Expects**: Hash | ||
**Default**: {} | ||
```ruby | ||
plugins = { | ||
'proxy' => {}, # empty options | ||
'autologin' => { | ||
'url' => 'http://demo.testfire.net/bank/login.aspx', | ||
'params' => 'uid=jsmith&passw=Demo1234', | ||
'check' => 'MY ACCOUNT' | ||
}, | ||
} | ||
p instance.call( "plugins.load", plugins ) | ||
#=> ["proxy", "autologin"] | ||
``` | ||
<a id="modules" href="#modules">Load the modules you want to run</a> | ||
-------------------------------------------- | ||
**Expects**: Array of Strings <br/> | ||
**Default**: [] | ||
```ruby | ||
# to load all modules | ||
# mods = [ '*' ] | ||
mods = [ 'xss', 'path_traversal' ] | ||
p instance.call( "modules.load", mods ) | ||
#=> ["xss", "path_traversal"] | ||
``` | ||
<a id="run" href="#run">Run the framework</a> | ||
------------------------- | ||
To run the framework: | ||
```ruby | ||
instance.call( "framework.run" ) | ||
``` | ||
### [Get the progress while busy](#run_busy) | ||
A handful of progress data can be requested by calling the | ||
```framework.progress_data``` method. | ||
This will include messages, statistics, discovered issues, status etc.<br/> | ||
See: | ||
[http://arachni.github.com/arachni/Arachni/RPC/Server/Framework.html#progress_data-instance_method](http://arachni.github.com/arachni/Arachni/RPC/Server/Framework.html#progress_data-instance_method) | ||
```ruby | ||
while instance.call( "framework.busy?" ) | ||
instance.call( "framework.progress_data" )['messages'].each do |out| | ||
type, msg = *out.to_a.first | ||
next if msg.empty? | ||
puts "#{type}: #{msg}" | ||
end | ||
sleep 0.5 | ||
end | ||
``` | ||
### <a id="run_pause_resume" href="#run_pause_resume">Pausing and Resuming</a> | ||
```ruby | ||
cnt = 0 | ||
while instance.call( "framework.busy?" ) | ||
instance.call( "service.output" ).each do |out| | ||
type, msg = *out.to_a.first | ||
next if msg.empty? | ||
puts "#{type}: #{msg}" | ||
end | ||
if cnt % 2 == 0 | ||
puts 'Pausing...' | ||
instance.call( "framework.pause!" ) | ||
else | ||
puts 'Resuming...' | ||
instance.call( "framework.resume!" ) | ||
end | ||
sleep 0.5 | ||
cnt += 1 | ||
end | ||
``` | ||
<a id="report" href="#report">Get the report</a> | ||
------------------------- | ||
To grab the results of the audit as a hash: | ||
```ruby | ||
instance.call( "framework.report" ) | ||
# to receive a YAML serialized hash -- may prevent type errors | ||
#instance.call( "framework.serialized_report" ) | ||
``` | ||
To grab the results of the audit as an AuditStore object: | ||
```ruby | ||
instance.call( "framework.auditstore" ) | ||
# to receive a YAML serialized auditstore | ||
#instance.call( "framework.serialized_auditstore" ) | ||
``` | ||
<a id="shutdown" href="#shutdown">Shutdown the server</a> | ||
-------------------------------- | ||
To completely shutdown the server: | ||
```ruby | ||
instance.call( "service.shutdown" ) | ||
``` | ||
<a id="cancel" href="#cancel">Cancelling the scan</a> | ||
------------------------------ | ||
In order to cancel a running scan it"s best to tell the Arachni instance | ||
to clean up after itself before forcing a shutdown.<br/> | ||
That way the framework and all running plugins will get a chance to | ||
register their results. | ||
```ruby | ||
instance.call( "framework.clean_up" ) | ||
report = instance.call( "framework.report" ) | ||
# or | ||
# report = instance.call( "framework.auditstore" ) | ||
instance.call( "service.shutdown" ) | ||
``` | ||
<a id="example" href="#example">A minimalistic example</a> | ||
---------------------------------- | ||
```ruby | ||
require 'arachni/rpc/pure' | ||
require 'pp' | ||
dispatcher = Arachni::RPC::Pure::Client.new( | ||
host: 'localhost', | ||
port: 7331 | ||
) | ||
instance_info = dispatcher.call( "dispatcher.dispatch" ) | ||
host, port = instance_info['url'].split( ':' ) | ||
instance = Arachni::RPC::Pure::Client.new( | ||
host: host, | ||
port: port, | ||
token: instance_info['token'] | ||
) | ||
begin | ||
opts = { | ||
'url' => 'http://demo.testfire.net', | ||
'audit_links' => true, | ||
'audit_forms' => true, | ||
'audit_cookies' => true, | ||
# 'link_count_limit' => 1, # uncomment this line for a quick scan | ||
} | ||
# instance.call( 'modules.load', ['xss'] ) | ||
instance.call( 'modules.load_all' ) # comment this line and uncomment the above line for a quick scan | ||
instance.call( 'opts.set', opts ) | ||
instance.call( 'framework.run' ) | ||
# | ||
# wait until the framework is finished | ||
# | ||
# you can also request a report at any point during the scan to get results | ||
# as they are logged but let's keep it simple for the example | ||
# | ||
print "Running." | ||
while instance.call( 'framework.busy?' ) | ||
sleep 1 | ||
print '.' | ||
end | ||
puts '[Done]' | ||
rescue | ||
puts | ||
puts 'Something bad happened.' | ||
instance.call( "framework.clean_up" ) | ||
ensure | ||
puts "Report:" | ||
puts '--------------' | ||
pp instance.call( 'framework.report' ) | ||
puts "Shutting down." | ||
instance.call( 'service.shutdown' ) | ||
end | ||
``` |