Re: Password Profiling and Password Mangling Libraries

Hi George

Dan proposed to include all profiling code to unpwdb (instead of a
separate pwdprofile library) [1] and I too believe it's better that
way

I wanted to separate the libraries to separate the responsibilities of the
libraries - one took care of loading username/password from files (and
keeping them as two separate tables) and provides the interface for
iterating through them, and the other one (pwdprofile) just solely looks at
storing interesting candidates (regardless of whether they are usernames or
passwords) and keeps its own list. However, I am fine with integrating it
into unpwdb, since unpwdb isn't really that big!

I'm not sure I understand the "pwdprofile.parse_html" method. Parsing
or decoding various formats is not part of the password profiler. It
is the script's responsibility to parse or decode this data and then
pass any interesting candidate phrases to profiler (e.g. with your
"add_phrase" method).

I was thinking from the perspective of httpspider, and how it might be
similar to CeWL. Many scripts use httpspider and I felt it didn't make
sense to implement it in httpspider.lua (since logic of password profiling
would then be in httpspider), and rather it would be best implemented in
pwdprofile (and we additionally provide a flag to activate it), and letting
httpspider simply pass the response to pwdprofile for processing.

Do let me know your thoughts on this!

Wai Tuck

On Mon, Jul 3, 2017 at 10:25 PM George Chatzisofroniou <[email protected]>
wrote:

> Dan proposed to include all profiling code to unpwdb (instead of a
> separate pwdprofile library) [1] and I too believe it's better that
> way.
>
> I'm not sure I understand the "pwdprofile.parse_html" method. Parsing
> or decoding various formats is not part of the password profiler. It
> is the script's responsibility to parse or decode this data and then
> pass any interesting candidate phrases to profiler (e.g. with your
> "add_phrase" method).
>
> [1]: http://seclists.org/nmap-dev/2017/q1/48
>
> George
>
>
> On Sun, Jul 2, 2017 at 12:22 AM, Wong Wai Tuck <[email protected]>
> wrote:
> > Hi list,
> >
> > I will be working on the password profiling library and password mangling
> > library in the coming weeks. This is a follow up from the improvements
> and
> > discussions first proposed by my mentor George [1] and the improvements
> > suggested by Dan [2]. Below shows the design plan for the two libraries.
> >
> >
> > pwdprofile.lua
> > The aim of this library is to keep track of possible username/password
> > candidates that are found from other scripts (e.g. http-title,
> http-passwd)
> > so that they can be used in the later part (for bruteforcing or to export
> > out to another program). pwdprofile.lua will also help to keep track of
> > which are the information gathering scripts that should run first, so
> that
> > they can simply be simply included into current brute scripts (e.g.
> > ftp-brute). By default, all information gathering scripts will log the
> > candidates to the library, and an optional argument will allow the user
> to
> > specify if he wishes export the list out to a csv file.
> >
> > Script Args
> > pwdprofile.export_file: the file to export data to, Default: nil
> > pwdprofile.disable: disables pwdprofile, no new candidates will be added,
> > Default false
> > pwdprofile.custom_file: the file to import candidates from, Default nil
> > pwdprofile.by_subnets: comma separated list of subnets that passwords
> will
> > be aggregated by, rather than on a global basis Default nil
> > pwdprofile.by_host: enables aggregation by host, rather than a global
> > aggregation, Default false
> >
> > Script Constants:
> > pwdprofile.STOP_WORDS_EN: a table of stopwords common in the English
> > language
> > pwdprofile.PWDPROFILE_SCRIPTS: a table of scripts that adds password
> > candidates to pwdprofile
> >
> > Script methods:
> > pwdprofile.add_word(host, keyword): saves a word verbatim to the password
> > profile table. As new words are added, if the export option is specified,
> > the word is also added to the file.
> >
> > pwdprofile.add_phrase(host, phrase, include_stop_words): parses a phrase
> for
> > unique words, optionally including stop words like 'the' [3]. Calls
> add_word
> > to add words. Default value for include_stop_words: false
> >
> > pwdprofile.parse_html(host, html, include_stop_words): parses the given
> html
> > page (as a string) for unique words, optionally including stop words like
> > 'the'.
> >
> > pwdprofile.get_candidates(host): returns the table containing the
> > username/password candidates that apply to the current host, given the
> > arguments, whether it is global (default), by subnet, or by host
> >
> >
> > pwdmangle.lua
> > The aim of this library is to provide an interface to stream possible
> > username password candidates into the iterator in unpwdb. By default, no
> > mangling is done so it streams whatever was originally in unpwdb. When
> rules
> > are provided, it will mangle the passwords provided by unpwdb and
> > pwdprofile, applying the rules given to generate additional words on the
> fly
> > through lazy evaluation. While I have yet to write something like this
> > before, I believe it can be done through a stateful class, where each
> brute
> > script will call unpwdb to create a new instance of pwdmangle, and stream
> > the password accordingly, so ideally the code for the brute scripts won't
> > change, but changes will be made to unpwdb such that the next suggested
> > password to use will be from pwdmangle.
> >
> > Script Arguments
> > pwdmangle.rules: file that specifies the rules to apply. Rules can
> include
> > our own (e.g. 1337 speak substitution) and those used by password
> crackers
> > [4]. In the new domain specific language we specify the rules in, we
> should
> > be able to combine lists, specify which particular list we are mangling
> (the
> > unpwdb provided one, the pwdprofile, or the custom wordlists via their
> index
> > in the comma separated arguments) or if we are mangling all lists
> globally.
> > Default nil
> > pwdmangle.wordlists: custom wordlist(s) provided by the user,comma
> > separated; Default nil
> > pwdmangle.ignore_default: ignores the default list given by unpwdb.
> Default:
> > false
> > pwdmangle.export: exports the passwordlist generated out to the file
> > provided. Useful for debugging passwords
> >
> > Class Mangler(host, unpwdb_passwds, wordlists, rules): constructor that
> > creates a new instance of Mangler based on the host, the password lists
> and
> > the rules specified in the arguments. The default instance simply does
> what
> > unpwdb does currently, and doesn't do any mangling. Otherwise, it
> creates a
> > stateful instance which keeps track of what the next password should be
> > based on what has been generated before and the rules given, and
> dynamically
> > generate them.
> >
> > mangler.get_next_password() returns the next password candidate based on
> the
> > current state of the object, or nil if there are no more passwords to
> > iterate through. The original passwords (given by unpwdb and pwdprofile
> are
> > prioritized first, followed by the rules applied).
> >
> > unpwdb
> > Modifications will be made to unpwdb to get passwords from pwdmangle,
> > passing in the original table so that if the user wants to apply the
> > mangling strategies to the original password list they can do so as well,
> > without breaking support for existing scripts users may have written.
> >
> > The dependencies will be as such
> > brute-script ----depends on----> unpwdb ---> pwdmangle --->pwdprofile
> >
> > Scripts affected by this change
> > discover/version/vuln/exploit scripts (adding useful information to
> > pwdprofile)
> > *-brute scripts (adding dependencies)
> > unpwdb
> >
> > Do let me know if you have any feedback for the design plan for the two
> > libraries!
> >
> > [1]: http://seclists.org/nmap-dev/2016/q2/46
> > [2]: http://seclists.org/nmap-dev/2017/q1/67
> > [3]: https://en.wikipedia.org/wiki/Stop_words
> > [4]: https://hashcat.net/wiki/doku.php?id=rule_based_attack
> >
> > With Regards
> > Wai Tuck
> >
> > _______________________________________________
> > Sent through the dev mailing list
> > https://nmap.org/mailman/listinfo/dev
> > Archived at http://seclists.org/nmap-dev/
>

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/