Skip to content
GitLab

RPC API · Changes

Page history
RPC-API guide: updated for v0.4.2 authored by Tasos Laskos's avatar Tasos Laskos
Hide whitespace changes
Inline Side-by-side
guides/developer/RPC-API.md
View page @ 7cc1a878
Version 0.4.1 ## Version 0.4.2
-------------
First of all, you should take a look at the
<a href="https://github.com/Arachni/arachni-rpc">Arachni-RPC</a>
protocol itself, especially its
<a href="https://github.com/Arachni/arachni-rpc/wiki">design specification</a>.
To provide functional examples of RPC interaction, I"ll be using the
<a href="https://github.com/Arachni/arachni-rpc-pure">pure Ruby client
implementation</a> of the Arachni-RPC protocol.
If you are interested in the full API you can browse through it
[here](http://arachni.github.com/arachni/Arachni/RPC/Server.html) -- the
accessible classes are Dispatcher, Instance and Framework.
In order to perform an audit you have to go through the following
straightforward steps:
1. [Connect to an Arachni RPC Dispatch server](#dispatcher_connect)
2. [Request an Arachni instance](#dispatch)
3. [Retrieve framework components (Optional)](#retrieve)
1. [Modules](#retrieve_modules)
2. [Plugins](#retrieve_plugins)
4. [Set the options you want](#options)
1. [Setting the URL (opts.url=)](#options_url)
2. [Audit links (opts.audit_links=)](#options_audit_links)
3. [Audit forms (opts.audit_forms=)](#options_audit_forms)
4. [Audit cookies (opts.audit_cookies=)](#options_audit_cookies)
5. [Audit headers (opts.audit_headers=)](#options_audit_headers)
6. [Link count limit (opts.link-count-limit=)](#options_link_count_limit)
7. [Exclude cookies (opts.exclude_cookies=)](#options_exclude_cookies)
8. [Exclude vectors (opts.exclude_vectors=)](#options_exclude_vectors)
9. [User agent (opts.user_agent=)](#options_user_agent)
10. [Exclude patterns (opts.exclude=)](#options_exclude)
11. [Include patterns (opts.include=)](#options_include)
12. [Cookie jar (opts.cookie_jar=)](#options_cookie_jar)
13. [HTTP request limit (opts.http-req-limit=)](#options_http_req_limit)
14. [Redundancy rules (opts.redundant=)](#options_redundant)
5. [Load the modules you want to run](#modules)
6. [Load the plugins you want to run](#plugins)
7. [Run the framework](#run)
1. [Get the progress while busy](#run_busy)
2. [Pausing and Resuming](#run_pause_resume)
8. [Get the report](#report)
9. [Shutdown the server](#shutdown)
10. [Cancelling the scan](#cancel)
### <a href="#example">A minimalistic example</a>
Setting up the test environment
-------------------------------
First of all, we"ll need to run an Arachni RPC Dispatcher to have
something to work and play with.
``` First of all, you should take a look at the [Arachni-RPC](https://github.com/Arachni/arachni-rpc)
$ arachni_rpcd protocol itself, especially its [design specification](https://github.com/Arachni/arachni-rpc/wiki).
Arachni - Web Application Security Scanner Framework v0.4.1dev
Author: Tasos "Zapotek" Laskos <[email protected]>
(With the support of the community and the Arachni Team.) To provide functional examples of RPC interaction, I'll be using the
[pure Ruby client implementation](https://github.com/Arachni/arachni-rpc-pure)
of the Arachni-RPC protocol.
Website: http://arachni-scanner.com The [RPC API of the Instances](http://arachni.github.com/arachni/Arachni/RPC/Server/Instance.html)
Documentation: http://arachni-scanner.com/wiki is well documented and contains all the info you'll need.
The only clarification required is about obtaining an Instance and maybe a couple
of examples to show you what controlling an Instance would look like.
Arachni - Web Application Security Scanner Framework v0.4.1dev 1. [Connect to an Arachni RPC Dispatch server](#dispatcher-connect)
Author: Tasos "Zapotek" Laskos <[email protected]> 2. [Request an Arachni instance](#dispatcher-dispatch)
3. [Connect to the Instance](#instance-connect)
4. [Talking to the service handler](#instance-service)
(With the support of the community and the Arachni Team.) ### Setting up the test environment
Website: http://arachni-scanner.com First of all, install the Arachni-RPC Pure client:
Documentation: http://arachni-scanner.com/wiki
```
gem install arachni-rpc-pure
```
I, [2012-08-28T05:29:39.412457 #23997] INFO -- System: RPC Server started. Now, we'll need to run an Arachni RPC Dispatcher to have something to work and play with.
I, [2012-08-28T05:29:39.412557 #23997] INFO -- System: Listening on localhost:1605
Arachni - Web Application Security Scanner Framework v0.4.1dev
Author: Tasos "Zapotek" Laskos <[email protected]>
(With the support of the community and the Arachni Team.) ```
$ arachni_rpcd
Arachni - Web Application Security Scanner Framework v0.4.2
Author: Tasos "Zapotek" Laskos <[email protected]>
Website: http://arachni-scanner.com (With the support of the community and the Arachni Team.)
Documentation: http://arachni-scanner.com/wiki
Website: http://arachni-scanner.com
Documentation: http://arachni-scanner.com/wiki
I, [2012-08-28T05:29:39.419214 #24001] INFO -- System: RPC Server started.
I, [2012-08-28T05:29:39.419314 #24001] INFO -- System: Listening on localhost:54160
[...lots of similar output...]
```
This is what happens when no options have been set; the default port is _7331_. Arachni - Web Application Security Scanner Framework v0.4.2
Author: Tasos "Zapotek" Laskos <[email protected]>
<a id="dispatcher_connect" href="#dispatcher_connect">Connect to an Arachni RPC Dispatch server</a> (With the support of the community and the Arachni Team.)
---------------------------------------------------------------
First of all, install the Arachni-RPC Pure client: Website: http://arachni-scanner.com
Documentation: http://arachni-scanner.com/wiki
I, [2012-08-28T05:29:39.412457 #23997] INFO -- System: RPC Server started.
I, [2012-08-28T05:29:39.412557 #23997] INFO -- System: Listening on localhost:1605
[...lots of similar output...]
``` ```
gem install arachni-rpc-pure
```
Simple as: This is what happens when no options have been set; the default port is `7331`.
### <a id="dispatcher-connect" href="#dispatcher-connect">Connect to an Arachni RPC Dispatch server</a>
```ruby ```ruby
require 'arachni/rpc/pure' require 'arachni/rpc/pure'
...@@ -115,18 +66,25 @@ dispatcher = Arachni::RPC::Pure::Client.new( ...@@ -115,18 +66,25 @@ dispatcher = Arachni::RPC::Pure::Client.new(
) )
``` ```
<a id="dispatch" href="#dispatch">Request an Arachni instance</a> ### <a id="dispatcher-dispatch" href="#dispatcher-dispatch">Request an Arachni instance</a>
----------------------------------------
```ruby ```ruby
# request for an instance to be dispatched # Request for an instance to be dispatched.
instance_info = dispatcher.call( 'dispatcher.dispatch' ) instance_info = dispatcher.call( 'dispatcher.dispatch' )
# =>
# {
# "token" => "bb5c94d2bd298cecf9da52421c0c0b71",
# "pid" => 26550,
# "port" => 62039,
# "url" => "localhost:62039",
# "owner" => "unknown",
# "birthdate" => 2013-03-30 04:34:26 +0200,
# "starttime" => 2013-03-30 04:36:24 +0200,
# "helpers" => {}
# }
``` ```
<a id="instance_connect" href="#instance_connect">Connect to the Arachni RPC instance</a> ### <a id="instance-connect" href="#instance-connect">Connect to the Arachni RPC instance</a>
--------------------------------------------------------
Simple as:
```ruby ```ruby
host, port = instance_info['url'].split( ':' ) host, port = instance_info['url'].split( ':' )
...@@ -140,607 +98,26 @@ instance = Arachni::RPC::Pure::Client.new( ...@@ -140,607 +98,26 @@ instance = Arachni::RPC::Pure::Client.new(
**In order to successfully authenticate yourself to the instance don't forget **In order to successfully authenticate yourself to the instance don't forget
to include the authentication token.** to include the authentication token.**
<a id="retrieve" href="#retrieve">Retrieve framework components (Optional)</a> ### <a id="instance-service" href="#instance-service">Talking to the service handler</a>
-----------------------------------------------------
This is strictly optional and is only useful when you"re developing an interface
and want to show the user all available components -- or in similar situations.
### <a id="retrieve_modules" href="#retrieve_modules">Modules (framework.lsmod)</a>
To retrieve all available modules:
```ruby
instance.call( 'framework.lsmod' )
```
Which will return something like:
```
[
[ 0] {
:name => "Code injection",
:description => "It tries to inject code snippets into the\n web application and assess whether or not the injection\n was successful.",
:elements => [
[0] "form",
[1] "link",
[2] "cookie",
[3] "header"
],
:author => [
[0] "Tasos \"Zapotek\" Laskos <[email protected]>"
],
:version => "0.1.6",
:references => {
"PHP" => "http://php.net/manual/en/function.eval.php",
"Perl" => "http://perldoc.perl.org/functions/eval.html",
"Python" => "http://docs.python.org/py3k/library/functions.html#eval",
"ASP" => "http://www.aspdev.org/asp/asp-eval-execute/",
"Ruby" => "http://en.wikipedia.org/wiki/Eval#Ruby"
},
:targets => [
[0] "PHP",
[1] "Perl",
[2] "Python",
[3] "ASP",
[4] "Ruby"
],
:issue => {
:name => "Code injection",
:description => "Arbitrary code can be injected into the web application\n which is then executed as part of the system.",
:tags => [
[0] "code",
[1] "injection",
[2] "regexp"
],
:cwe => "94",
:severity => "High",
:cvssv2 => "7.5",
:remedy_guidance => "User inputs must be validated and filtered\n before being evaluated as executable code.\n Better yet, the web application should stop evaluating user\n inputs as any part of dynamic code altogether.",
:remedy_code => "",
:metasploitable => "unix/webapp/arachni_php_eval"
},
:mod_name => "code_injection",
:path => "/home/zapotek/workspace/arachni/modules/audit/code_injection.rb"
},
[ 1] {
:name => "PathTraversal",
:description => "It injects paths of common files (/etc/passwd and boot.ini)\n and evaluates the existence of a path traversal vulnerability\n based on the presence of relevant content in the HTML responses.",
:elements => [
[0] "form",
[1] "link",
[2] "cookie",
[3] "header"
],
:author => [
[0] "Tasos \"Zapotek\" Laskos <[email protected]>"
],
:version => "0.2.6",
:references => {
"OWASP" => "http://www.owasp.org/index.php/Path_Traversal",
"WASC" => "http://projects.webappsec.org/Path-Traversal"
},
:targets => [
[0] "Unix",
[1] "Windows",
[2] "Tomcat"
],
:issue => {
:name => "Path Traversal",
:description => "The web application enforces improper limitation\n of a pathname to a restricted directory.",
:tags => [
[0] "path",
[1] "traversal",
[2] "injection",
[3] "regexp"
],
:cwe => "22",
:severity => "Medium",
:cvssv2 => "4.3",
:remedy_guidance => "User inputs must be validated and filtered\n before being used as a part of a filesystem path.",
:remedy_code => "",
:metasploitable => "unix/webapp/arachni_path_traversal"
},
:mod_name => "path_traversal",
:path => "/home/zapotek/workspace/arachni/modules/audit/path_traversal.rb"
},
[...and many more...]
]
```
### <a id="retrieve_plugins" href="#retrieve_plugins">Plugins (framework.lsplug)</a>
To retrieve all available plugins
```ruby
instance.call( 'framework.lsplug' )
```
```
[
[ 0] {
:name => "Resolver",
:description => "Resolves vulnerable hostnames to IP addresses.",
:author => [
[0] "Tasos \"Zapotek\" Laskos <[email protected]>"
],
:tags => [
[0] "ip address",
[1] "hostname"
],
:version => "0.1.1",
:plug_name => "resolver",
:path => "/home/zapotek/workspace/arachni/plugins/defaults/resolver.rb",
:options => []
},
[ 1] {
:name => "Health map",
:description => "Generates a simple list of safe/unsafe URLs.",
:author => [
[0] "Tasos \"Zapotek\" Laskos <[email protected]>"
],
:version => "0.1.3",
:plug_name => "healthmap",
:path => "/home/zapotek/workspace/arachni/plugins/defaults/healthmap.rb",
:options => []
},
[...and many more...]
]
```
<a id="options" href="#options">Set the options you want</a>
------------------------------------
You can do that in 2 ways, either set them all at once or one at a time.
To set them all at once:
```ruby
# you can also use Strings as keys
opts = {
url: 'http://demo.testfire.net',
audit_links: true,
audit_forms: true,
audit_cookies: true,
audit_headers: true,
link_count_limit: 1,
user_agent: "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-us) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3Safari/5",
}
p instance.call( 'opts.set', opts )
#=> true
```
You can use any of the available options that will be presented later in the ```opts``` hash.
### <a id="options_url" href="#options_url">Setting the URL (opts.url=)</a>
**Expects**: String <br/>
**Default**: <n/a>
To set the URL of the site you want to audit:
```ruby
p instance.call( 'opts.url=', 'http://demo.testfire.net' )
#=> "http://demo.testfire.net"
```
### <a id="options_audit_links" href="#options_audit_links">Audit links (opts.audit_links=)</a>
**Expects**: Boolean <br/>
**Default**: false
To audit link elements:
```ruby
p instance.call( 'opts.audit_links=', true )
#=> true
p instance.call( 'opts.audit', :links )
#=> true
p instance.call( 'opts.audit?', :links )
#=> true
p instance.call( 'opts.dont_audit', :links )
#=> true
p instance.call( 'opts.audit?', :links )
#=> false
```
### <a id="options_audit_forms" href="#options_audit_forms">Audit forms (opts.audit_forms=)</a>
**Expects**: Boolean <br/>
**Default**: false
To audit form elements:
```ruby
p instance.call( 'opts.audit_forms=', true )
#=> true
p instance.call( 'opts.audit', :forms )
#=> true
p instance.call( 'opts.audit?', :forms )
#=> true
p instance.call( 'opts.dont_audit', :forms )
#=> true
p instance.call( 'opts.audit?', :forms )
#=> false
```
### <a id="options_audit_cookies" href="#options_audit_cookies">Audit cookies (opts.audit_cookies=)</a>
**Expects**: Boolean <br/>
**Default**: false
To audit cookies:
```ruby
p instance.call( 'opts.audit_cookies=', true )
#=> true
p instance.call( 'opts.audit', :cookies )
#=> true
p instance.call( 'opts.audit?', :cookies )
#=> true
p instance.call( 'opts.dont_audit', :cookies )
#=> true
p instance.call( 'opts.audit?', :cookies )
#=> false
```
### <a id="options_audit_headers" href="#options_audit_headers">Audit headers (opts.audit_headers=)</a>
**Expects**: Boolean <br/>
**Default**: false
To audit headers:
```ruby
p instance.call( 'opts.audit_headers=', true )
#=> true
p instance.call( 'opts.audit', :headers )
#=> true
p instance.call( 'opts.audit?', :headers )
#=> true
p instance.call( 'opts.dont_audit', :headers )
#=> true
p instance.call( 'opts.audit?', :headers )
#=> false
```
### <a id="options_link_count_limit" href="#options_link_count_limit">Link count limit (opts.link_count_limit=)</a>
**Expects**: Integer <br/>
**Default**: infinite
To limit how many pages will be crawled and audited:
```ruby
p instance.call( 'opts.link_count_limit=', 1 )
#=> 1
```
###<a id="options_exclude_cookies" href="#options_exclude_cookies">Exclude cookies (opts.exclude_cookies=)</a>
**Expects**: Array of Strings <br/>
**Default**: []
To exclude cookies from the audit process:
```ruby
p instance.call( 'opts.exclude_cookies=', [ 'sessionid', 'some_auth_cookie' ] )
#=> ["sessionid", "some_auth_cookie"]
```
### [Exclude vectors (opts.exclude_vectors=)](#options_exclude_vectors)
**Expects**: Array of Strings <br/>
**Default**: []
To exclude input vectors from the audit process:
```ruby
p instance.call( 'opts.exclude_vectors=', [ 'password', 'csrf_token' ] )
#=> ["password", "csrf_token"]
```
### [User agent (opts.user_agent=)](#options_user_agent)
**Expects**: String <br/>
**Default**: Arachni/\<version\>
To set the user agent:
```ruby
p instance.call( 'opts.user_agent=', 'FooBar/0.1' )
#=> "FooBar/0.1"
```
### <a id="options_exclude" href="#options_exclude">Exclude patterns (opts.exclude=)</a>
**Expects**: Array of patterns (String or Regexp) <br/>
**Default**: []
To set an exclude rule:
```ruby
p instance.call( "opts.exclude=", ["do_not_follow"] )
#=> [/do_not_follow/]
```
URLs that match any of the patterns in the array will be ignored.
### <a id="options_include" href="#options_include">Include patterns (opts.include=)</a>
**Expects**: Array of patterns (String or Regexp) <br/>
**Default**: []
To set an include rule:
```ruby
p instance.call( "opts.include=", ["only_follow_me"] )
#=> [/only_follow_me/]
```
Only URLs that match any of the patterns in the array will be followed and audited.
### <a id="options_cookie_jar" href="#options_cookie_jar">Cookie jar (opts.cookie_jar=)</a>
**Expects**: Hash <br/>
**Default**: {}
To set the cookie jar:
```ruby
cookies = {
"userid" => 1,
"sessionid" => "fdfdfDDfsdfszdf"
}
p instance.call( "opts.cookie_jar=", cookies )
#=> {"userid"=>1, "sessionid"=>"fdfdfDDfsdfszdf"}
```
### <a id="options_http_req_limit" href="#options_http_req_limit">HTTP request limit (opts.http_req_limit=)</a>
**Expects**: Integer <br/>
**Default**: 20
To limit how many concurrent HTTP request are sent:
```ruby
p instance.call( "opts.http_req_limit=", 20 )
#=> 20
```
**Note**: If your scan seems unresponsive try lowering the limit. <br>
**Warning**: Given enough bandwidth and a high limit it could cause a DoS. Be
careful when setting this option too high, don"t kill your server.
### <a id="options_redundant" href="#options_redundant">Redundancy patterns (opts.redundant=)</a>
**Expects**: Array of Hashes <br/>
**Default**: []
```ruby
redundant = { "follow_me_3_times" => 3, /follow_me_5_times/ => 5 }
p instance.call( "opts.redundant=", redundant )
#=> {/follow_me_3_times/=>3, /follow_me_5_times/=>5}
```
<a id="plugins" href="#plugins">Load the plugins you want to run</a>
--------------------------------------------
**Expects**: Hash
**Default**: {}
```ruby
plugins = {
'proxy' => {}, # empty options
'autologin' => {
'url' => 'http://demo.testfire.net/bank/login.aspx',
'params' => 'uid=jsmith&passw=Demo1234',
'check' => 'MY ACCOUNT'
},
}
p instance.call( "plugins.load", plugins )
#=> ["proxy", "autologin"]
```
<a id="modules" href="#modules">Load the modules you want to run</a>
--------------------------------------------
**Expects**: Array of Strings <br/>
**Default**: []
```ruby
# to load all modules
# mods = [ '*' ]
mods = [ 'xss', 'path_traversal' ]
p instance.call( "modules.load", mods )
#=> ["xss", "path_traversal"]
```
<a id="run" href="#run">Run the framework</a>
-------------------------
To run the framework: To make things easier, the [service](http://arachni.github.com/arachni/Arachni/RPC/Server/Instance.html)
handler of the RPC Instance exposes a number of methods which provide the most
commonly used functionality.
That way, you won't have to talk directly to the more specialized, and often
more complicated, system components (like the Framework, Module manager, Plugin
manager, etc.).
```ruby Let's see how one would go about performing a few calls.
instance.call( "framework.run" )
```
### [Get the progress while busy](#run_busy)
A handful of progress data can be requested by calling the ```framework.progress_data``` method.
This will include messages, statistics, discovered issues, status etc.<br/>
See: [Framework#progress_data](http://rubydoc.info/github/Arachni/arachni/Arachni/RPC/Server/Framework.html#progress_data-instance_method)
```ruby
while instance.call( "framework.busy?" )
instance.call( "framework.progress_data" )['messages'].each do |out|
type, msg = *out.to_a.first
next if msg.empty?
puts "#{type}: #{msg}"
end
sleep 0.5
end
```
### <a id="run_pause_resume" href="#run_pause_resume">Pausing and Resuming</a>
```ruby
cnt = 0
while instance.call( "framework.busy?" )
instance.call( "service.output" ).each do |out|
type, msg = *out.to_a.first
next if msg.empty?
puts "#{type}: #{msg}"
end
if cnt % 2 == 0
puts 'Pausing...'
instance.call( "framework.pause!" )
else
puts 'Resuming...'
instance.call( "framework.resume!" )
end
sleep 0.5
cnt += 1
end
```
<a id="report" href="#report">Get the report</a>
-------------------------
To grab the results of the audit as a hash:
```ruby ```ruby
instance.call( "framework.report" ) # Calls #list_modules on the 'service' handler.
instance.call( 'service.list_modules' )
# to receive a YAML serialized hash -- may prevent type errors
#instance.call( "framework.serialized_report" )
``` ```
To grab the results of the audit as an AuditStore object: To call a server-side method with parameters you simply pass those parameters to
the `#call` method of the RPC client, like so:
```ruby ```ruby
instance.call( "framework.auditstore" ) # Calls #progress on the 'service' handler with a parameter.
instance.call( 'service.progress', without: :stats )
# to receive a YAML serialized auditstore
#instance.call( "framework.serialized_auditstore" )
```
<a id="shutdown" href="#shutdown">Shutdown the server</a>
--------------------------------
To completely shutdown the server:
```ruby
instance.call( "service.shutdown" )
```
<a id="cancel" href="#cancel">Cancelling the scan</a>
------------------------------
In order to cancel a running scan it"s best to tell the Arachni instance
to clean up after itself before forcing a shutdown.<br/>
That way the framework and all running plugins will get a chance to
register their results.
```ruby
instance.call( "framework.clean_up" )
report = instance.call( "framework.report" )
# or
# report = instance.call( "framework.auditstore" )
instance.call( "service.shutdown" )
```
<a id="example" href="#example">A minimalistic example</a>
----------------------------------
```ruby
require 'arachni/rpc/pure'
require 'pp'
dispatcher = Arachni::RPC::Pure::Client.new(
host: 'localhost',
port: 7331
)
instance_info = dispatcher.call( "dispatcher.dispatch" )
host, port = instance_info['url'].split( ':' )
instance = Arachni::RPC::Pure::Client.new(
host: host,
port: port,
token: instance_info['token']
)
begin
opts = {
'url' => 'http://demo.testfire.net',
'audit_links' => true,
'audit_forms' => true,
'audit_cookies' => true,
# 'link_count_limit' => 1, # uncomment this line for a quick scan
}
# instance.call( 'modules.load', ['xss'] )
instance.call( 'modules.load_all' ) # comment this line and uncomment the above line for a quick scan
instance.call( 'opts.set', opts )
instance.call( 'framework.run' )
#
# wait until the framework is finished
#
# you can also request a report at any point during the scan to get results
# as they are logged but let's keep it simple for the example
#
print "Running."
while instance.call( 'framework.busy?' )
sleep 1
print '.'
end
puts '[Done]'
rescue
puts
puts 'Something bad happened.'
instance.call( "framework.clean_up" )
ensure
puts "Report:"
puts '--------------'
pp instance.call( 'framework.report' )
puts "Shutting down."
instance.call( 'service.shutdown' )
end
``` ```