Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Sunday, July 20, 2014

My first attempt at using AWS EC2

There are lots of cloud hosting services out there.   AWS is one of the most popular (if not the most popular), so I decided to set myself up with a free account so I could learn how to use it.   This blog post covers my initial experiences.
  • Signing up is very easy, just go to aws.amazon.com.   I signed in with my personal Amazon account, and created an AWS account.
  • I will probably be using EC2, and RDS -  An EC2 instance (VM) to host server-side web applications (Java) and RDS for the database.    I will probably use EBS as well, so I can have some durable filesystem storage for the EC2 instance.
  • I started with the "Basic" free tier.    You need to enter your CC information though, in case you go over the limitations of the free tier.   Since I'm mostly just going to be creating VMs for learning, mostly likely I won't be keeping too many instances running.

The free tier

Currently the AWS free usage tier gives you the following for one year:
  •  EC2 (virtual machines) - 750 hours/month on a 't2.micro' instance that is Amazon Linux, RHEL, or SLES
  • EBS (file system storage) - 30GB, 2 million I/O ops, 1G of snapshot storage
  • RDS (Relational db) - 750 hours/month on a 'micro' instance, 20G of storage, 20G of backup, 10M I/O ops
 See http://aws.amazon.com/free

What's a t2.micro instance?

T2 is Amazons instance type that is optimized for 'burstable performance'.   A t2.micro instance has:
  • 1 CPU and 1G of RAM.
  • Only EBS for durable storage (i.e. anything not on EBS will be lost when the machine is shut down).
750 hours per month?   Should I start and stop my instances?

You probably shouldn't start and stop instances too often.   The billing granularity is hourly, so if you start an instance, you might as well keep it running for an hour.    If you stop an instance, you might as well keep it stopped for at least an hour.

Also, if you start and stop an instance three times in an hour, Amazon will bill you for three hours.   So, you need to think about whether you really need to shut down or not.   This makes sense because Amazon probably doesn't want everybody to be constantly starting and stopping machines all the time.

See this page for more.

It is also a good idea to enable billing alerts.

Launching an Instance

Go to the AWS console, click on EC2.   Click 'Launch Instance'.
  1. Chose a machine image - Make sure you check the 'Free tier only' box if you want to stay in the free tier.   I chose Amazon Linux.
  2. Choose an instance type - t2.micro is the only free tier instance type, so I chose that.
  3. Configure instance - leave the defaults
  4. Add storage - leave the defaults
  5. Tag instance - leave the defaults
  6. Configure Security Group - Since I'm doing this for the first time, I created a new security group called "Administrators".   I chose 'My IP' for SSH access.   Should be good enough for today, and I suppose that I can change that access rule via the AWS console later to add new IP addresses.  Click 'Review and launch'

    Boot from General Purpose (SSD) prompt: keep the default choice.  Click Next.
  7. Review - This should all look okay, so just go ahead and launch it.

    Create a new key pair: Select 'Create a new key pair' and enter the key pair name.   You'll need to download the private key (.pem file) and store it somewhere.   I put my in a Google Drive folder so I could get to it later.

Connect to the new Linux instance with SSH


See this page for Windows/Putty and this page for Linux/OSX ssh.

You'll need the private key, the instance id, and the public DNS address of the instance.

Amazon Linux


This linux distro is in the Red Hat family - it uses yum and rpm.   Many packages are available to install.   I saw that mine had a Java 7 JRE installed, and that the yum repo had Tomcat 7 available, as well as MySQL and other things.

What's next?

  • Set up Tomcat, enable HTTPS access from the outside.
  • Set up MySQL on RDS - Connect Tomcat to MySQL.
  • Look into making my own machine images (AMIs) that have everything pre-installed and set up.
Once I get Tomcat->MySQL going, hopefully I can begin installing webapps to see how well the t2.micro instance works.    If it works well, I might consider moving my home wiki to AWS.

 I may also consider doing the same thing with Open Shift, to compare and contrast the costs and ease of use.







Tuesday, April 22, 2014

Upgrading Fedora - Notes

A few notes on upgrading Fedora installations.



Fresh Install 

Probably the safest way to get a working upgrade is to back up any home directories or important configurations and go with a fresh installation.

Upgrades often leave undesirable configurations in home directories (GNOME configs, for example).  This often leads to strange desktop / display issues that can't easily by found or fixed.

You are not using OSX here.   Migrating settings and applications may or may not work.  :)

1. Create a USB Stick

On Fedora 19, these instructions didn't work for me.   Here is what I ended up doing.  Get a USB stick that doesn't have anything important on it.
  1. Download the ISO image.
  2. Insert the USB stick.
  3. Start the Disks application and select the USB drive in the left panel. 
  4. Unmount the USB disk filesystem if it is mounted. 
  5. Up at the top of the right panel, click on the gear icon and select Restore Disk Image.
  6. Select the downloaded ISO image file, and click Start Restoring.... 

2. Boot using the USB Stick, complete the installation

Shut down the machine, and re-start.   If you need to, use the BIOS to select the USB as the boot drive. 

Go through the install process.   Best to dedicate a HDD to the install, that way you can boot from that drive via BIOS boot selector if you want multiple OS's on your computer without too much hassle.   In my case, I've got a dual boot workstation, with a HDD dedicated to booting Linux.   I use the BIOS boot drive selector to boot up Fedora instead of WinDoze. 

NOTE: I've found that EZbcd doesn't play nice with UEFI boot partitions that Fedora 20 installs.    Best to just use the BIOS to select a boot disk.

 

 

 

Using FedUp

WARNING, THIS DOESN'T ALWAYS WORK.   Almost every time I've done this, there were some strange after-effects with GNome at least.

For newer versions of Fedora (newer than 17), FedUp with the network upgrade is the way to go:

$ sudo yum install fedup
$ sudo yum update fedup fedora-release
$ sudo fedup --network 20

Where 20 is the version you want to upgrade to.  Fedup will automatically reboot the system when it's done downloading everything.


The Fedora site says: "Prior to Fedora 17, the DVD/ISO/USB Drive option is recommended."

Yeah, well... what they really mean is, that FedUp will probably get you something that boots and runs some things, but you may discover later on that many settings are just plain broken.


Sunday, January 12, 2014

Replacing a bad hard drive in a ZFS pool - Linux/zfs-fuse

Thought I'd re-post this here, for convenience. I've got a home-brew NAS server that is running Fedora, zfs-fuse, and CIFS. The situation:
  • "Disk Utility" reports that drive /dev/sde has many bad sectors.
  • zpool status shows a degraded state for the main pool.  The drive is listed in the main pool by it's id.
    # zpool status -v
      pool: nasdata
     state: DEGRADED
    status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
    action: Replace the device using 'zpool replace'.
       see: http://www.sun.com/msg/ZFS-8000-4J
     scrub: resilver completed after 0h28m with 0 errors on Sat Feb 23 23:54:32 2013
    config:
    
        NAME                                      STATE     READ WRITE CKSUM
        nasdata                                   DEGRADED     0     0     0
          raidz1-0                                DEGRADED     0     0     0
            disk/by-id/ata-ST31500541AS_5XW0PDZ1  ONLINE       0     0     0
            disk/by-id/ata-ST31500541AS_5XW0PZJQ  ONLINE       0     0     0
            disk/by-id/ata-ST31500541AS_6XW1MKZZ  UNAVAIL      0   193     6  experienced I/O failures
            disk/by-id/ata-ST31500541AS_6XW1KRR9  ONLINE       0     0     0
    
    errors: No known data errors
    
Well that pretty much sums it up. No data errors in the array itself, but the disk is unavailable. Here's the process for replacing it:

  1. Tell zfs to take the disk offline:
    # zpool offline nasdata /dev/disk/by-id/ata-ST31500541AS_6XW1MKZZ
    

    Note that I'm using /dev/disk/by-id here. This is because that is how it is listed in the pool. 
  2. Shut the machine down.
  3. Add the new disk.  I also removed the failing disk because it was causing problems during POST.
    NOTE: REMEMBER TO LABEL YOUR DISKS! This really helps when the time comes to replace them! 
  4. Start the machine up.
  5. Tell zfs about the new disk:
    # zpool replace nasdata /dev/disk/by-id/ata-ST31500541AS_6XW1MKZZ /dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ90BA09450
    

    Note: I had to use the disk IDs because the pool set itself up that way in the first place (I had switched the drives to a new SATA card).
  6. Immediately ZFS begins replacing the disk:
    # zpool status 
      pool: nasdata
     state: DEGRADED
    status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
    action: Wait for the resilver to complete.
     scrub: resilver in progress for 0h0m, 0.00% done, 2695h59m to go
    config:
        NAME                                                 STATE     READ WRITE CKSUM
        nasdata                                              DEGRADED     0     0     0
          raidz1-0                                           DEGRADED     0     0     0
            disk/by-id/ata-ST31500541AS_5XW0PDZ1             ONLINE       0     0     0
            disk/by-id/ata-ST31500541AS_5XW0PZJQ             ONLINE       0     0     0
            replacing-2                                      DEGRADED     0     0     0
              disk/by-id/ata-ST31500541AS_6XW1MKZZ           OFFLINE      0   193     6
              disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ90BA09450  ONLINE       0     0     0  2.34M resilvered
            disk/by-id/ata-ST31500541AS_6XW1KRR9             ONLINE       0     0     0
    errors: No known data errors
    

    Now hopefully this won't take 2695 hours to complete! :) Later on the status goes down to 11h. Okay, that's doable. 
  7. Several hours later, the new drive is incorporated into the pool:
    # zpool status -v
      pool: nasdata
     state: ONLINE
     scrub: resilver completed after 9h2m with 0 errors on Sun Feb 24 10:30:21 2013
    config:
            NAME                                               STATE     READ WRITE CKSUM
            nasdata                                            ONLINE       0     0     0
              raidz1-0                                         ONLINE       0     0     0
                disk/by-id/ata-ST31500541AS_5XW0PDZ1           ONLINE       0     0     0
                disk/by-id/ata-ST31500541AS_5XW0PZJQ           ONLINE       0     0     0
                disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ90BA09450  ONLINE       0     0     0  916G resilvered
                disk/by-id/ata-ST31500541AS_6XW1KRR9           ONLINE       0     0     0
    errors: No known data errors
So that's it. While it was re-slivering, the ZFS filesystem was completely available. How nice! My SMB/CIFS shares were working just fine.