Re: Password Profiling and Password Mangling Libraries

Dan proposed to include all profiling code to unpwdb (instead of a
separate pwdprofile library) [1] and I too believe it's better that
way.

I'm not sure I understand the "pwdprofile.parse_html" method. Parsing
or decoding various formats is not part of the password profiler. It
is the script's responsibility to parse or decode this data and then
pass any interesting candidate phrases to profiler (e.g. with your
"add_phrase" method).

[1]: http://seclists.org/nmap-dev/2017/q1/48

George


On Sun, Jul 2, 2017 at 12:22 AM, Wong Wai Tuck <[email protected]> wrote:
> Hi list,
>
> I will be working on the password profiling library and password mangling
> library in the coming weeks. This is a follow up from the improvements and
> discussions first proposed by my mentor George [1] and the improvements
> suggested by Dan [2]. Below shows the design plan for the two libraries.
>
>
> pwdprofile.lua
> The aim of this library is to keep track of possible username/password
> candidates that are found from other scripts (e.g. http-title, http-passwd)
> so that they can be used in the later part (for bruteforcing or to export
> out to another program). pwdprofile.lua will also help to keep track of
> which are the information gathering scripts that should run first, so that
> they can simply be simply included into current brute scripts (e.g.
> ftp-brute). By default, all information gathering scripts will log the
> candidates to the library, and an optional argument will allow the user to
> specify if he wishes export the list out to a csv file.
>
> Script Args
> pwdprofile.export_file: the file to export data to, Default: nil
> pwdprofile.disable: disables pwdprofile, no new candidates will be added,
> Default false
> pwdprofile.custom_file: the file to import candidates from, Default nil
> pwdprofile.by_subnets: comma separated list of subnets that passwords will
> be aggregated by, rather than on a global basis Default nil
> pwdprofile.by_host: enables aggregation by host, rather than a global
> aggregation, Default false
>
> Script Constants:
> pwdprofile.STOP_WORDS_EN: a table of stopwords common in the English
> language
> pwdprofile.PWDPROFILE_SCRIPTS: a table of scripts that adds password
> candidates to pwdprofile
>
> Script methods:
> pwdprofile.add_word(host, keyword): saves a word verbatim to the password
> profile table. As new words are added, if the export option is specified,
> the word is also added to the file.
>
> pwdprofile.add_phrase(host, phrase, include_stop_words): parses a phrase for
> unique words, optionally including stop words like 'the' [3]. Calls add_word
> to add words. Default value for include_stop_words: false
>
> pwdprofile.parse_html(host, html, include_stop_words): parses the given html
> page (as a string) for unique words, optionally including stop words like
> 'the'.
>
> pwdprofile.get_candidates(host): returns the table containing the
> username/password candidates that apply to the current host, given the
> arguments, whether it is global (default), by subnet, or by host
>
>
> pwdmangle.lua
> The aim of this library is to provide an interface to stream possible
> username password candidates into the iterator in unpwdb. By default, no
> mangling is done so it streams whatever was originally in unpwdb. When rules
> are provided, it will mangle the passwords provided by unpwdb and
> pwdprofile, applying the rules given to generate additional words on the fly
> through lazy evaluation. While I have yet to write something like this
> before, I believe it can be done through a stateful class, where each brute
> script will call unpwdb to create a new instance of pwdmangle, and stream
> the password accordingly, so ideally the code for the brute scripts won't
> change, but changes will be made to unpwdb such that the next suggested
> password to use will be from pwdmangle.
>
> Script Arguments
> pwdmangle.rules: file that specifies the rules to apply. Rules can include
> our own (e.g. 1337 speak substitution) and those used by password crackers
> [4]. In the new domain specific language we specify the rules in, we should
> be able to combine lists, specify which particular list we are mangling (the
> unpwdb provided one, the pwdprofile, or the custom wordlists via their index
> in the comma separated arguments) or if we are mangling all lists globally.
> Default nil
> pwdmangle.wordlists: custom wordlist(s) provided by the user,comma
> separated; Default nil
> pwdmangle.ignore_default: ignores the default list given by unpwdb. Default:
> false
> pwdmangle.export: exports the passwordlist generated out to the file
> provided. Useful for debugging passwords
>
> Class Mangler(host, unpwdb_passwds, wordlists, rules): constructor that
> creates a new instance of Mangler based on the host, the password lists and
> the rules specified in the arguments. The default instance simply does what
> unpwdb does currently, and doesn't do any mangling. Otherwise, it creates a
> stateful instance which keeps track of what the next password should be
> based on what has been generated before and the rules given, and dynamically
> generate them.
>
> mangler.get_next_password() returns the next password candidate based on the
> current state of the object, or nil if there are no more passwords to
> iterate through. The original passwords (given by unpwdb and pwdprofile are
> prioritized first, followed by the rules applied).
>
> unpwdb
> Modifications will be made to unpwdb to get passwords from pwdmangle,
> passing in the original table so that if the user wants to apply the
> mangling strategies to the original password list they can do so as well,
> without breaking support for existing scripts users may have written.
>
> The dependencies will be as such
> brute-script ----depends on----> unpwdb ---> pwdmangle --->pwdprofile
>
> Scripts affected by this change
> discover/version/vuln/exploit scripts (adding useful information to
> pwdprofile)
> *-brute scripts (adding dependencies)
> unpwdb
>
> Do let me know if you have any feedback for the design plan for the two
> libraries!
>
> [1]: http://seclists.org/nmap-dev/2016/q2/46
> [2]: http://seclists.org/nmap-dev/2017/q1/67
> [3]: https://en.wikipedia.org/wiki/Stop_words
> [4]: https://hashcat.net/wiki/doku.php?id=rule_based_attack
>
> With Regards
> Wai Tuck
>
> _______________________________________________
> Sent through the dev mailing list
> https://nmap.org/mailman/listinfo/dev
> Archived at http://seclists.org/nmap-dev/
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/