SA Architect Best Practices

[ThomasJones1]

UPDATED - 07/20/2016 - Tom J

1. Teach the customer to fish - empower them, give them confidence, teach them how the system works.
2. Determine what the customer needs are from the SA or ECAT perspective.
3. Determine if the customer needs fit within the boundaries of the proposed solution.
4. Ask the right SA or ECAT questions.  Wrong questions will waste your time and the customer's time.
5. Architect for the future.  This will save valuable time and assist in the customer success and overall satisfaction.
6. Be realistic.  Take on the low hanging fruit first.  Get the customer to 80% using existing rules.  The other 20% will be for customized work.

General lessons.

1. Not all hosts are equal - this seems obvious but I see this problem all the time. Customers should identify critical systems such as domain controllers, web servers, etc.  Our job, to implement a solution that protects these vital systems from the other non-critical system.
2. Deploy VLCs locally.  Customers should not be pulling remote logs from across geographic locations - to avoid latency issues, excessive bandwidth usage, etc.  Example, do not pull logs from Hawaii to New York over the WAN... deploy a VLC in Hawaii and pull the logs to the VLC then send from the Hawaii VLC to New York in one stream, not many streams.
3. Size does matter - over engineer the solution when you can but keep the architecture simple and logical.  The more complicated a solution becomes the more difficult it is to maintain it (KISS principal).  Again, this translates into customer success and satisfaction.
4. Be involved with the customer.  As engineers we typically avoid the small talk and get to business.  Start a light conversation with the customer to build rapport.  You'll be amazed at what the customer will share with you regarding his/her environment and enable you to obtain a deeper understanding.
5. Avoid making customer adjustments that only you know about, such as custom scripting that will impact the environment - this does not include monitoring and other general tasks such as cron.  Work within the confines of the product... avoid "creative modifications" to circumvent a problem with the system because chances upgrades, service patches, etc may impact it negatively.  Escalate the problem so engineering can fix it.

The observations above are my short list and some items may be a little controversial but my question is what are your best practices?

Thanks

Tom J

[SoumyajitDhara]

This is really valuable and guideline information, especially for those who working as Resident Engineer.

[ShawnOstapuk]

I agree with most of it, but I think #2 may not apply to many enterprises. Depending on your companies requirements, particularly if you have to collect UDP syslog, a VLC is not a high availability device. There is no clustering, reboots, upgrades, and service restarts can cause log collection downtime, rabbitmq issues or plugin issues can cause the process to crash and often they are not configured to handle a huge spike in syslog.

It can be beneficial to build out a few high availability syslog front-ends servers in a (relatively) few number of primary locations and configure with clustering, store and forward, advanced filtering and then forward on to your VLCs using the Z-connector template.

This also gives you the flexibility to forward some or all syslog events to other teams or systems without having to configure forwarding on the decoders.

Every company is going to be different and deploying VLCs in every location may not be the right answer for them.

[ThomasJones1]

Thanks Shawn for thoughtful reply.  I agree it is customer dependent and if the customer doesn't mind paying for the WAN traffic and can live with the connectivity issues then it is their call.