Skip to content

LOPSA East 2014 Review

LOPSA East 2014 wrapped up a few minutes ago and I wanted to record my thoughts while they were fresh. This is the first LOPSA East that I have been to, as in the past there wasn’t enough content geared towards me (Windows admin) to warrant a trip down from Canada.

This year I checked out the schedule pretty early on and I saw more Windows material as well as a lot of business-related workshops (presentation skills, management, business navigation, etc.). With the various tracks and recognizing the names of some of presenters, I decided to try it out.

By the end of the first day I was convinced that I made a great decision. I was able to attend Tom Limoncelli’s ‘Evil Genius’ class that focused on dealing with difficult people. The tips he provided within a context familiar to Sys Admins made so much sense. Tom was very interactive and open to any questions or comments the room had. Plus we all got our Evil Genius certificate of completion : )

For the afternoon track, I went into Scott Cromar’s terrific talk about making the transition from techie to manager. We discussed many different issues including delegation, role changes, discipline, as well as how to do deal with being promoted and being your ex co-worker’s boss. I have a copy of his book that I plan on digging into shortly. On top of that, I was fortunate enough to sit next to him at lunch today and we were able to get a literal round-table discussion going on best management practices, and discuss some challenges that others at the table faced.

The second day (today) was filled with a great in-depth primer of Active Directory on server 2012, delivered by Mark Marra. This was a great ‘overview, but technical’ class where we reviewed the core features of Active Directory Domain Services and what each component does. Mark was fantastic at taking questions and making sense of situations where seemingly stupid rules might apply.

After lunch I was able to grab a talk on Meraki hardware (which turned into a great peer discussion), and a talk about presentation skills for technical employees. I made plenty of notes during this talk and hope to put them to good use very soon.

To cap the talks off, I was able to grab Tom’s 5 Time Management Tips talk followed by an exclusive preview of his upcoming book, The Practice of Cloud Administration (due out later this year). He was gracious enough to give us a sneak peak at the content, as well as to ask for our thoughts and opinions on the cover and title. You can let him know what you think about the title by visiting

All in all, this has to have been one of the best conferences I have attended. The amount of skill and passion that everyone brought, along with how easy it was to have productive talks with anyone, made this whole conference great. Can’t wait until LOPSA East 2015.

Initial thoughts on Veeam Backup & Replication

Veeam Backup & ReplicationRecently I have been looking to revamp our backup scheme as frankly, what we had wasn’t working (I won’t get into this now – a bit of a mess). One of my key requirements was that I could do file level restores on my VMs without having to restore the whole VMDK.

After talking to various vendors and playing around with a few choices, I ‘stumbled’ upon Veeam. Now, I knew the name Veeam from when their product originally launched, but I hadn’t really looked to see what they were up to lately.

Veeam Backup & Replication is their current flagship product, and as the name insinuates, it backs up and replicates VMWare machines. But the real question was how well does it do it.

Quite well.

OK, I’ll elaborate. Veeam Backup & Replication has a few different options for their back ups – full, incremental or reverse incremental. Their reverse incremental essentially takes incremental back ups (to save time and space) but it writes the data necessary to ensure that the latest job can do a full restore (i.e. you only need the full job and the last reverse incremental to get back to current – if you want to go further back then you need the incrementals from the newest to that time).

Speed wise, this thing is excellent. Most jobs only take minutes as it just does a differential. Dedupe is also built in to help decrease the data set size.

On the restore side, individual files can be restored from OSs (Windows / Linux) as can MS SQL DBs. Exchange mailboxes and Active Directory objects. I’ll touch on these at a later date.

Lastly, what made this a no-brainer for me was the price. It licensed per physical processor, so what would have cost me $20K + for software agents with other vendors, is under $3K  for three processor licenses (two for main VM host and one for replicated host).

I’ll do a follow up post with a few more details (including replication), but so far I am quite happy with it.

When error messages don’t mean what they say (I’m looking at you 0×80070070)

I had one of our VPs stop by to chat with me today and he mentioned that a laptop that his division uses for presentations has been ‘messing up’ again. I say ‘again’ because a few months ago it would sporadically get caught in perpetual reboots (although I couldn’t reproduce it). That time I ended up running a chkdsk /R and giving it back; after it apparently did it again I ended up just wiping/reloading it – that was several months ago.

Sometimes Windows' error messages are like reading a different language

Sometimes Windows' error messages are like reading a different language

Back to the story …. So I get the laptop turn it on, and see that it spends about a minute at the Windows XP splash screen and then BSODs (too quick for me to see what it actually says). ‘Hmmm’ I say to myself – do I spend the time to actually try and figure out what is going on, or do I take 15 minutes to re-image the laptop. Seeing as how time is a precious commodity at work these days, I opted for the re-imaging approach (of course I confirmed that there was no data needed off of the drive).

So thanks to my handy-dandy Windows Deployment Services setup that I have, I am able to reload fully patched Windows XP SP3 images in minutes; did I mention that these images also include all of our standard corporate software (e.g. Cisco VPN client, Office, etc)?

So I start going along my merry way with a PXE boot and choose my images, watch them copy files/expand, and then *BAM* good old 0×80070070. My first thought was that maybe it was the wrong image file …. Although that shouldn’t matter at this stage. So I rebooted, tried another image file and the same thing … 0×80070070.

A quick Google brings up a lot of Vista install problems with this particular error message, but not a whole lot relating to Windows Deployment Services. Since WDS uses the same environment as Vista to deploy itself I figured that there was a good chance that there would be some overlap. So apparently 0×80070070 means low disk space – however the laptop I was trying to image had 18GB free …. Even though that should really matter as the partition should be formatted anyways.

The fix turned out to be delete the partitions (C: and D: ), create a new partition and voila – WDS imaging in all its goodness. I ended up just booting off a Windows XP disk, deleting/creating the partition and the rebooting.

So long story short, 0×80070070 may not be quite as it states but at least it seems to be in the general area.

Tagged , , , , , ,

Sever Security

Network EquipmentMost of us know that security for our systems isn’t just a one step process – in fact there can be many steps involved with getting a system secure. One of the most obvious steps is also one that I see most often ignored: physical security.

When I first started working for my current employer, they stashed the servers in a small corner room (maybe 10 x 10) and the door would sometimes be left open, sometimes closed, all depending on who was in there last. Realistically there should have been only two people allowed access to that room (three if you count me), but yet it was essentially a spare office for anyone who walked by.

On my first day when I was first showed the server room, I remember remarking how hot it was – there was no ventilation or A/C[1]. Later that day I found out that the door didn’t even have a lock on it. I also remarked about that, but it didn’t seem to be of much concern.

A couple of weeks later I walked in there to take care of something, and lo and behold someone (outside of the IT department) was in there taking a phone call. I was a little surprised to say the least. In fact she was sitting at a small desk in there making notes. I was just thankful that didn’t decide to log on to the server so that she could play solitaire or something [2].

It was shortly after that I really pushed to have a lock installed; if I was going to be responsible for these servers, then I wanted to make sure nobody was screwing around with them. I showed the powers that be how easily I can get into a system if I had physical access to it: I could pull out a hard drive, I could install a key logger, I could use a Linux boot disk to change the Administrator password, or I could even have changed the time in the BIOS of the domain controller – that would have really pooched things.

Long story short – if someone has physical access to your computer, you better have damn good encryption on the drive, otherwise it’s game over.

Note: This post from The Daily WTF reminded me of my experience.

[1] My request to get a portable A/C unit was finally approved after one of the systems overheated and shut down on a weekend.

[2] To make matters worse, the other two people in the department where consistently leaving the servers logged in under the Administrator accounts. That was just utterly ridiculous, but don’t get me started on that.

Tagged ,

Are you being spied on?

Your securityA few days ago a report was published by the security firm Cyber-Ark which stated that 1 in 3 senior IT professionals snoops/spies on their colleagues. I have no doubt that this does happen, but I do have to raise a few questions about this report.

First of all let me preface this by saying that I have never spied or co-workers or snooped through their stuff. Quite frankly, I barely have time for my own life, let alone time to go through other people’s email and files. I also tried to stress to co-workers that I can appreciate their privacy and I will uphold it the best I can [1].

Anyways, my first beef with the report is the sample size: 300 people. How many hundreds of thousands of people are in a position to spy on co-workers (from an IT standpoint)? I would imagine quite a few, yet a sample of 300 people creates a blanket statement.

Secondly, what size shops were the respondents from? Were they small shops (less than 50 people) or larger places (hundreds? thousands?). This seems relevant to me because I know that most smaller shops lack policies when it comes to this type of stuff. Further to that, most larger companies usually (hopefully) will have auditing and monitoring systems in place to make sure nobody is doing or seeing anything that they shouldn’t.

Finally I have to question whether this is just an attempt to spread FUD. I mean come on, a security company publishes a report saying that people chances are your IT guy is snooping on you.

Unfortunately I can’t help but feel that this report will cast IT pros in a bad light. Let’s face it; the average user will see this and automatically wonder if their IT staff are in the 1 in 3 category instead of lumping them in with the 2 in 3 category.

[1] Of course there are times when their privacy has to be secondary, like if there are legal issues or if a manager has a valid reason to access staff data.


Gmail to the rescue

When trying to solve some mail server problems, one of the most useful troubleshooting tools that I use (aside from good old telnet) has to be free email service providers. Whether it is Hotmail, Yahoo or Gmail, having a free email account set up with them can greatly help when it comes to figuring out where the problem lies.

Quite often if somebody says to me ‘hey, is our email down’, one of the first things I will do is send an email from my corporate account to a personal account. This will tell me right away if our server is accepting connections internally, it will let me know that our DNS is working properly (it would have to in order to resolve the recipients domain MX) and it will let me know that it is sending mail to external recipients.

I usually follow this up with a reply to the test message to let me know that our DNS records are up and running correctly and that our mail server is accepting incoming connections from outside the network.

Of course there could be a slew of other problems in between (whether it be spam/content filters, storage issues, firewall issues, etc), but having an accessible external email address sure helps narrow down where the potential problem is.

Even if the message doesn’t get through to the external account, at least now you have a known message that you can trace through the log files.

Tagged ,

TechSoup – A must for non-profits

As I mentioned in my last post, the organization which I work for is a non-profit entity whose budget comes from membership fees. Like most non-profits, every dollar really has to stretch as far as it can, and it is usually hard to get an OK from management to go out and spend X thousand dollars on software licensing. I was in this situation a few years ago, right around when Windows XP SP2 came out.

I first started working for the organization in 2003, and most of the systems were running Windows 2000 Professional. By the time Windows XP SP2 was released I had started looking into performing a company wide upgrade, but I was quickly shot down due to the cost of licensing. I talked to a Microsoft partner and we went through a bunch of different licensing options from retail to Open Volume, but I just couldn’t the number low enough to get the OK.

I knew that unless I could lower the cost, the only way the systems would ever get upgraded would be through buying new systems. Even at that, it would be a fight to spend the extra cash on upgrading from Windows XP Home to Windows XP Professional.

After some searching around, I found a true gem: TechSoup. TechSoup’s purpose is to be a middleman between non-profits and software companies. They administer relations between companies and take care of a lot of the leg work; in return they are able to provide software (and hardware in some cases) to non-profit organizations at huge discounts. All they charge is an administrative fee. Windows XP Professional upgrade, for example, costs $8 a license.

So if you are a non-profit looking to cut costs while still having access to great software, make sure you go to TechSoup and see what they have. It’s not just Microsoft software there either. Other vendors include Adobe, Cisco, and Symantec.

Tagged , ,

About Me

Hi there and welcome to my tiny corner on the web. My name is Matt and I am a Systems Administrator for a small business (under 50 employees). We’re not a typical business however. Out of all our employees, over half are remote. Most of them have their own office which means that I technically have to support about 20 locations.

Another unique aspect of my job is that we are a non-profit, member based organization which throws all sorts of curve balls our way. If members aren’t happy, they cancel their membership and our budget shrinks.

As I start to post more and more, I am sure that details will come out about some of the unique situations I run into, but that’s part of the reason why I decided to start blogging. My intent for this site is that it will be place where I can record my thoughts on things that I run across daily and if that happens to help someone else along the way, all the better.