Re: [p4] Can Perforce replication achieve these results: would this hypothetical topology work in reality?

Posted on behalf of forum user 'Matt Janulewicz'.

When you start talking about a writeable replica, it sounds like 'edge
server' to me. And yes, there is a bunch of metadata that is unique to an
edge (which you should back up.) Off the top of my head:

db.have
unload depot
spec depot
labels (unless you set them to global)
shelves (unless you set them to global)

There's probably more associated with features we don't use. In any
case, we've been running on a commit->edge architecture for almost three
years now, with five edge servers spread throughout the world. It works really
well for us. We haven't yet deployed a build-type server for alleviating
stress on our most busy edge, but we're working on it. (We'd still
deploy it as an edge since some of our builds alter files.)

When it comes to performance/delay, to expand on what p4rfong talks about, just
note that the live journal and root db on any Perforce server will be written to
at the same time. You very much should not ignore the advice about filesystems
in various KB's. Absolutely, positively do not put the db and the journal on
the same spindles (or ssd's or flash.) Separate hardware is key there for
when floods of data come pouring in. As mentioned by p4rfong, you'll mostly
notice it during very large transactions, like say someone submitting 40,000
files at once. I think there will be at least 3x that many lines being written
to the db during that transaction and sometimes that takes a while, so fast disk
for both the journal and db is essential. Not so much for the library files, if
you want someplace to go cheap.

There's also some presentations floating around from the Perforce
conferences. I can't seem to find them on perforce.com any more and the
wayback machine is not friendly to the Merge 2016 page, so suffice to say that
Windows should be avoided if you're concerned with performance, and use ZFS
on Linux at your own risk. It's pretty solid but not the best performer
(improving with every release, though.) It's hard to go wrong with just
plain 'ol xfs.

Overall, not accounting for the random huge transaction, in our case the
day-to-day flow of the journal is easily managed by a somewhat beefy machine. We
have over 800 users, hundreds of submits a day, db's that range from about
400 GB to 800 GB, 10's of thousands of daily requests overall, but our daily
rotated journals are generally somewhere between 1 and 10 GB, depending on how
busy the edge servers are. Worst case, that's a 10 GB file that is growing
at a relatively consistent rate during any 24 hour period, so it's not
really hammering the server every second of every day, it just needs to absorb
those rather large transactions once in a while and maintain a steady flow of
input throughout the day.

'p4 pull -lj' is very useful for figuring out if your hardware is up to
the task once you deploy a replica. Example output:

$ p4 pull -lj Current replica journal state is: Journal 4800, Sequence
5419276. Current master journal state is: Journal 4800, Sequence 5419479. The
statefile was last modified at: 2017/12/15 11:12:07. The replica server time is
currently: 2017/12/15 11:12:07 -0800 PST   
The difference between the times for 'statefile was last modified at'
and 'replica server time is currently' is your 'journal delay',
as I call it. The Sequence numbers should be pretty close to each other but for
us they are rarely identical during the work day. That represents the byte
location that the journal has 'absorbed' on the replica vs how big the
journal is on the master. In other words, the difference is how many bytes the
replica is lagging behind.

We parse out those four data points and throw them into our monitoring system
(Zabbix), then calculate that journal delay every minute, we're only really
concerned if the delay is what we deem 'long', and usually a minute or
so is when I'd start to get worried that something awful is happening.
It's also nice to have history and charts so if something happens after
hours you can correlate it with a transaction (you're using the command
structured log, right? If not, you should.)

Armed with the journal delay history and the command log, it's usually
pretty easy to point to a user doing something abusive or more likely our build
farm going nuts and creating an inadvertent DDoS attack on our servers.



--
Please click here to see the post in its original format:
  http://forums.perforce.com/index.php?/topic/5546-can-perforce-replication-achieve-these-results-would-this-hypothetical-topology-work-in-reality
_______________________________________________
perforce-user mailing list  -  [email protected]
http://maillist.perforce.com/mailman/listinfo/perforce-user