Thursday, December 27. 2012
Samba 4.0.0
Today's entry is about one of the most important open source projects: Samba. This project is the standard Windows interoperability suite of programs for Linux and Unix and just a few days ago the first stable version of Samba 4 series was released. This milestone is enormously important for the project, cos this version can act as an Active Directory (AD) Domain Component (DC), supporting AD logon (any version of current windows client can now join a Samba domain). After nine years of developing and 21 alpha releases, 8 betas and 6 release candidates the final stable version 4.0.0 was made public. I think the Samba project was the first open source project that I was interested in and I started to follow. Being the icon against closed software at first times, the project has changed very little while the world around it was under transformation. When I saw that Microsoft was even collaborating with the team I finally realized that positions had changed in closed/open software relationships. Now I am (much to my sorrow) less informed about this great project but, celebrating that event, I am going to dedicate the entry to test new Samba 4.0.0.
Mainly I followed the Samba 4 howto from its wiki page.
First a Wheezy box was installed and the following extra packages were added:
# apt-get install build-essential libacl1-dev libattr1-dev libblkid-dev libgnutls-dev libreadline-dev python-dev python-dnspython gdb pkg-config libpopt-dev libldap2-dev dnsutils libbsd-dev attr krb5-user docbook-xsl bind9 libkrb5-dev git libpam0g-dev libcups2-dev ntp ldap-util quota acl
Some of the packages can be not necessary but the real requirements are explained here. Cos I wanted to test acls, quotas and so on I changed the root file system this way:
# cat /etc/fstab | grep "ext4" UUID=39c4ea16-e59c-4d2a-8f76-c9fd3179b59c / ext4 errors=remount-ro,usrquota,grpquota 0 1 # mount | grep "on / " /dev/disk/by-uuid/39c4ea16-e59c-4d2a-8f76-c9fd3179b59c on / type ext4 (rw,relatime,errors=remount-ro,user_xattr,barrier=1,data=ordered,usrquota,grpquota)
I just added quotas cos default options give to the file system user extended attributes.
Then the compressed tarball was downloaded, compiled and installed:
# tar zxvf samba-4.0.0.tar.gz # ./configure --enable-debug --enable-selftest # make # make test --quick # make install
I tried first to compile it with MIT kerberos but it seems that Samba 4 needs internal Heimdal implementation in order to work as a Domain Component (trying to compile it with MIT results like compiling with the --without-ad-dc option). The default installation directory is /usr/local/samba.
After that the domain should be provisioned. I set the --dns-backend=BIND9_DLZ option, that means that complex AD and DNS interaction is made using a Dynamically Loadable Zones (DLZ) module (this option needs bind compiled with DLZ support and current 9.8.1 package in Wheezy supports it), and --use-rfc2307, this option make the winbind daemon get the uid and guid from RFC2307 attributes in AD (at first I thought Samba would pull uidNumber and gidNumber at user creation but no, you have to do that as it is said in this forum).
# /usr/local/samba/bin/samba-tool domain provision --domain=KVM --realm=KVM.TEST --server-role=dc --adminpass=Kiosko_00 --dns-backend=BIND9_DLZ --use-rfc2307 Looking up IPv4 addresses Looking up IPv6 addresses No IPv6 address will be assigned Setting up share.ldb Setting up secrets.ldb Setting up the registry Setting up the privileges database Setting up idmap db Setting up SAM db Setting up sam.ldb partitions and settings Setting up sam.ldb rootDSE Pre-loading the Samba 4 and AD schema Adding DomainDN: DC=kvm,DC=test Adding configuration container Setting up sam.ldb schema Setting up sam.ldb configuration data Setting up display specifiers Adding users container Modifying users container Adding computers container Modifying computers container Setting up sam.ldb data Setting up well known security principals Setting up sam.ldb users and groups Setting up self join Adding DNS accounts Creating CN=MicrosoftDNS,CN=System,DC=kvm,DC=test Creating DomainDnsZones and ForestDnsZones partitions Populating DomainDnsZones and ForestDnsZones partitions See /usr/local/samba/private/named.conf for an example configuration include file for BIND and /usr/local/samba/private/named.txt for further documentation required for secure DNS updates Setting up sam.ldb rootDSE marking as synchronized Fixing provision GUIDs A Kerberos configuration suitable for Samba 4 has been generated at /usr/local/samba/private/krb5.conf Setting up fake yp server settings Once the above files are installed, your Samba4 server will be ready to use Server Role: active directory domain controller Hostname: samba NetBIOS Domain: KVM DNS Domain: kvm.test DOMAIN SID: S-1-5-21-2145774160-2038213957-1596523949
Then the dynamic zone for AD is configured in bind:
# cat /etc/bind/named.conf // This is the primary configuration file for the BIND DNS server named. // // Please read /usr/share/doc/bind9/README.Debian.gz for information on the // structure of BIND configuration files in Debian, BEFORE you customize // this configuration file. // // If you are just adding zones, please do that in /etc/bind/named.conf.local include "/etc/bind/named.conf.options"; include "/etc/bind/named.conf.local"; include "/etc/bind/named.conf.default-zones"; include "/usr/local/samba/private/named.conf";
The samba daemon can now be started:
# /usr/local/samba/sbin/samba
Let's do some tests. List the services running in Samba:
# /usr/local/samba/bin/smbclient -L localhost -U% Domain=[KVM] OS=[Unix] Server=[Samba 4.0.0] Sharename Type Comment --------- ---- ------- netlogon Disk sysvol Disk IPC$ IPC IPC Service (Samba 4.0.0) Domain=[KVM] OS=[Unix] Server=[Samba 4.0.0] Server Comment --------- ------- Workgroup Master --------- -------
Perform a ls command in the netlogon:
# /usr/local/samba/bin/smbclient //localhost/netlogon -UAdministrator%'Kiosko_00' -c 'ls' Domain=[KVM] OS=[Unix] Server=[Samba 4.0.0] . D 0 Tue Dec 25 13:40:02 2012 .. D 0 Tue Dec 25 13:40:12 2012 45559 blocks of size 65536. 9267 blocks available
Check that some names in the DNS are working:
# host -t SRV _kerberos._udp.kvm.test. _kerberos._udp.kvm.test has SRV record 0 100 88 samba.kvm.test. # host -t SRV _ldap._tcp.kvm.test. _ldap._tcp.kvm.test has SRV record 0 100 389 samba.kvm.test. # host -t A samba.kvm.test. samba.kvm.test has address 192.168.122.13
As AD is up and running, all the typical info can be requested at empty base, data like contexts, capabilities, controls and so on (I did not check if it is the same data as the information answered by a real AD).
# ldapsearch -LLL -h localhost -p 389 -D "Administrator@KVM.TEST" -w Kiosko_00 -b "" -s base dn= dn: configurationNamingContext: CN=Configuration,DC=kvm,DC=test defaultNamingContext: DC=kvm,DC=test rootDomainNamingContext: DC=kvm,DC=test schemaNamingContext: CN=Schema,CN=Configuration,DC=kvm,DC=test subschemaSubentry: CN=Aggregate,CN=Schema,CN=Configuration,DC=kvm,DC=test supportedCapabilities: 1.2.840.113556.1.4.800 supportedCapabilities: 1.2.840.113556.1.4.1670 supportedCapabilities: 1.2.840.113556.1.4.1791 supportedCapabilities: 1.2.840.113556.1.4.1935 supportedCapabilities: 1.2.840.113556.1.4.2080 supportedLDAPVersion: 2 supportedLDAPVersion: 3 vendorName: Samba Team (http://samba.org) isSynchronized: TRUE dsServiceName: CN=NTDS Settings,CN=SAMBA,CN=Servers,CN=Default-First-Site-Name ,CN=Sites,CN=Configuration,DC=kvm,DC=test serverName: CN=SAMBA,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configu ration,DC=kvm,DC=test dNSHostName: samba.kvm.test ldapServiceName: kvm.test:samba$@KVM.TEST currentTime: 20121226195225.0Z supportedControl: 1.2.840.113556.1.4.841 supportedControl: 1.2.840.113556.1.4.319 supportedControl: 1.2.840.113556.1.4.473 supportedControl: 1.2.840.113556.1.4.1504 supportedControl: 1.2.840.113556.1.4.801 supportedControl: 1.2.840.113556.1.4.801 supportedControl: 1.2.840.113556.1.4.805 supportedControl: 1.2.840.113556.1.4.1338 supportedControl: 1.2.840.113556.1.4.529 supportedControl: 1.2.840.113556.1.4.417 supportedControl: 1.2.840.113556.1.4.2064 supportedControl: 1.2.840.113556.1.4.1413 supportedControl: 1.2.840.113556.1.4.1413 supportedControl: 1.2.840.113556.1.4.1413 supportedControl: 1.2.840.113556.1.4.1413 supportedControl: 1.2.840.113556.1.4.1413 supportedControl: 1.2.840.113556.1.4.1339 supportedControl: 1.2.840.113556.1.4.1340 supportedControl: 1.2.840.113556.1.4.1413 supportedControl: 1.2.840.113556.1.4.1341 namingContexts: DC=kvm,DC=test namingContexts: CN=Configuration,DC=kvm,DC=test namingContexts: CN=Schema,CN=Configuration,DC=kvm,DC=test namingContexts: DC=DomainDnsZones,DC=kvm,DC=test namingContexts: DC=ForestDnsZones,DC=kvm,DC=test supportedSASLMechanisms: GSS-SPNEGO supportedSASLMechanisms: GSSAPI supportedSASLMechanisms: NTLM highestCommittedUSN: 3803 domainFunctionality: 2 forestFunctionality: 2 domainControllerFunctionality: 4 isGlobalCatalogReady: TRUE
The AD information is stored using the new LDB database engine developed by the team too. In the /usr/local/samba/private/ directory there are several LDB files which store different data (all the AD entries, DNS data and so on). This is very important cos now in Samba 4 this storage engine is the only one supported (in previous releases other repositories can be used for some specific parts).
Then the /etc/resolv.conf was configured to search over the local DNS, the system kerberos was setup to attack Samba 4 and ticketing was tested:
# cat /etc/resolv.conf domain kvm.test nameserver 19.2168.122.13 # cat /usr/local/samba/share/setup/krb5.conf [libdefaults] default_realm = "KVM.TEST" dns_lookup_realm = false dns_lookup_kdc = true # kinit administrator@KVM.TEST Password for administrator@KVM.TEST: Warning: Your password will expire in 41 days on Tue Feb 5 13:40:10 2013 # klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: administrator@KVM.TEST Valid starting Expires Service principal 25/12/2012 14:57 26/12/2012 00:57 krbtgt/KVM.TEST@KVM.TEST renew until 26/12/2012 14:57
To complete the domain provision the DNS should be configured to update the AD zone. This step is done just adding the following option to bind files:
# cat /etc/bind/named.conf.options options { directory "/var/cache/bind"; ... // samba 4 tkey-gssapi-keytab "/usr/local/samba/private/dns.keytab"; }; # /usr/local/samba/sbin/samba_dnsupdate --all-names
As the last step, NTP was configured in the samba machine too (it is known that kerberos and AD needs proper time synchronization with clients to proper logon). I configured the server to let the machines in the network to question this host:
restrict 192.168.122.0 mask 255.255.255.0
Then you can test the machine is synchronized against the NTP pool:
# ntpq -n -p localhost remote refid st t when poll reach delay offset jitter ============================================================================== 84.77.40.132 130.206.3.166 2 u 15 64 37 75.035 4.127 7.337 +147.83.123.133 193.67.79.202 2 u 5 64 77 57.778 3.094 5.415 +46.17.142.10 158.227.98.15 2 u 7 64 77 70.119 5.455 7.780 -84.88.69.32 193.67.79.202 2 u 42 64 77 82.793 -8.122 10.070
After that a share was added in the /usr/local/samba/etc/smb.conf file:
[data] path = /data writable = yes browsable = yes
And I created a folder (0 is administrator uid and 100 the Domain Users group gid, see later how to know those numbers):
# mkdir /data # chown 0:100 /data
After that some users and groups were created:
# /usr/local/samba/bin/samba-tool user add ricky New Password: Retype Password: User 'ricky' created successfully # /usr/local/samba/bin/samba-tool user add santi New Password: Retype Password: User 'santi' created successfully # /usr/local/samba/bin/samba-tool group add group1 Added group group1 # /usr/local/samba/bin/samba-tool group add group2 Added group group2 # /usr/local/samba/bin/samba-tool group add group3 Added group group3 # /usr/local/samba/bin/samba-tool group addmembers group1 ricky Added members to group group1 # /usr/local/samba/bin/samba-tool group addmembers group2 santi Added members to group group2 # /usr/local/samba/bin/samba-tool group addmembers group3 ricky,santi Added members to group group3
The samba-tool command has several functions when Samba 4 acts like a DC. The users can then be obtained using normal ldapsearch against the AD:
# ldapsearch -h localhost -p 389 -D "Administrator@KVM.TEST" -LLL -w Kiosko_00 -b "DC=KVM,DC=TEST" samaccountname=ricky dn: CN=ricky,CN=Users,DC=kvm,DC=test objectClass: top objectClass: person objectClass: organizationalPerson objectClass: user cn: ricky instanceType: 4 whenCreated: 20121225213834.0Z whenChanged: 20121225213834.0Z uSNCreated: 3769 name: ricky objectGUID:: JIRbSHnZXEqKYAUuabOZPg== badPwdCount: 0 codePage: 0 countryCode: 0 badPasswordTime: 0 lastLogoff: 0 lastLogon: 0 primaryGroupID: 513 objectSid:: AQUAAAAAAAUVAAAAUOrlf0WtfHmtBSlfUAQAAA== accountExpires: 9223372036854775807 logonCount: 0 sAMAccountName: ricky sAMAccountType: 805306368 userPrincipalName: ricky@kvm.test objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=kvm,DC=test pwdLastSet: 130009451140000000 userAccountControl: 512 uSNChanged: 3771 memberOf: CN=group1,CN=Users,DC=kvm,DC=test memberOf: CN=group3,CN=Users,DC=kvm,DC=test distinguishedName: CN=ricky,CN=Users,DC=kvm,DC=test
Any user (or group) is assigned to a UID and a SID, so there is a map between Unix and Windows information. This mapping is stored again in a LDB file, in that case the idmap.ldb. With the current configuration the Windows user is just an UID, not a real Unix user, until we setup winbind but this part is not done in the entry. That linking information can be checked using wbinfo command:
# /usr/local/samba/bin/wbinfo --user-info=ricky KVM\ricky:*:3000016:100::/home/KVM/ricky:/bin/false # /usr/local/samba/bin/wbinfo --name-to-sid=ricky S-1-5-21-2145774160-2038213957-1596523949-1104 SID_USER (1)
And they can be searched (Samba 4 gives ldb* commands to attack directly against the LDB files):
# /usr/local/samba/bin/ldbsearch -H /usr/local/samba/private/idmap.ldb objectsid=S-1-5-21-2145774160-2038213957-1596523949-1104 dn: CN=S-1-5-21-2145774160-2038213957-1596523949-1104 cn: S-1-5-21-2145774160-2038213957-1596523949-1104 objectClass: sidMap objectSid: S-1-5-21-2145774160-2038213957-1596523949-1104 type: ID_TYPE_BOTH xidNumber: 3000016 distinguishedName: CN=S-1-5-21-2145774160-2038213957-1596523949-1104
Finally you can create the user using LDAP directly against the AD port (I wanted to check that for provisioning purposes), the user is more or less added like in any other AD. The only problem I had was setting the password, but it was because it should be in UTF16 and quoted (it seems that samba, and maybe AD I do not know, uses three different password attributes unicodePwd, userPassword and clearTextPassword). Here it is a little java to change the password. As I commented before the RFC2307 (posix information) should be assigned after the creation (which is really nasty in my opinion).
In all the previous configurations of Samba I had seen, they used the ldap backend (user and groups were stored in a compatible LDAP server). But it seems that now the software does not support or recommend it. I suppose that implementing a LDAP server (AD is basically a LDAP directory server) over another LDAP server is not a good idea. But previous architecture let to construct a central / corporate directory that can be the user repository for all the company applications, now this idea can only be implemented using Samba as the common user repository (obviously this is not the fault of Samba team, it was the target of Microsoft when introduced AD, Samba is only giving the chance of not using Microsoft software).
After that a windows 2008R2 was installed and joined to the Samba domain. In order to do that a fixed address was configured in the same network: Start→Control Panel→Network and Internet→View Network Status and Tasks→Local Area Network. Remember that the DNS server should be the Samba one.
Then the windows box was configured to be in time against the Samba NTP following this information. And finally I changed the host to join to the Samba domain. My Computer→Properties→Change Settings→Change (it requests an Administrator user and password for the task).
Once the windows machine was rebooted I could login as domain Administrator. As him the share can be restricted using normal security (Properties→Security→Advanced), three directories were created and full permissions for the so-named group were assigned. All of that worked smoothly.
For example in the folder group3 those permissions were used:
As the file system uses user_xattr (I did not force that, but that is another option of domain provision and smb.conf file --use-xattrs) the permissions are assigned using ACLs, which can be checked in the system (remember wbinfo command can be used to map GIDs with Samba groups):
# getfacl /data/group3 # file: data/group3 # owner: 3000000 # group: users user::rwx group::--- group:users:--- group:3000000:rwx group:3000021:rwx mask::rwx other::--- default:user::rwx default:user:3000000:rwx default:group::--- default:group:users:--- default:group:3000000:rwx default:group:3000021:rwx default:mask::rwx default:other::---
The final test I did was assigning file system quotas to the root device. Cos the usrquota and grpquota were previously set we can assign the quota directly to user ricky:
# edquota -u 3000016 Disk quotas for user 3000016 (uid 3000016): Filesystem blocks soft hard inodes soft hard /dev/disk/by-uuid/39c4ea16-e59c-4d2a-8f76-c9fd3179b59c 8 100 200 1 0 0
After that I logged in in the windows box and created several files in the group1 directory. After the quota was reached a not enough space error is shown.
And the repquota command shows it is full (user ricky is using more than the 100 blocks soft limit):
# repquota / *** Report for user quotas on device /dev/disk/by-uuid/39c4ea16-e59c-4d2a-8f76-c9fd3179b59c Block grace time: 00:00; Inode grace time: 00:00 Block limits File limits User used soft hard grace used soft hard grace ---------------------------------------------------------------------- root -- 2088512 0 0 60007 0 0 daemon -- 68 0 0 4 0 0 man -- 1844 0 0 139 0 0 libuuid -- 4 0 0 1 0 0 Debian-exim -- 28 0 0 6 0 0 statd -- 12 0 0 3 0 0 ricky -- 21536 0 0 6 0 0 bind -- 16 0 0 5 0 0 ntp -- 12 0 0 3 0 0 #3000008 -- 64 0 0 8 0 0 #3000016 +- 104 100 200 none 12 0 0 #3000000 -- 8 0 0 1 0 0 #3000017 -- 8 0 0 1 0 0
As you have seen the entry explains the basics of Samba 4, a simple installation and some file serving testing (ACLs and quotas), I did not even setup a second DC or migrated a previous Samba 3. But some years ago I knew a lot about Samba and I wanted to refresh my knowledge with the new version. If Samba is going to be used as an AD DC, things have changed a lot. Now the software provides a complete AD/LDAP implementation made over their own LDB storage and, in general, it should be treated like any other Microsoft AD. It was a little disappointment at first but then I realized that it is the main goal that Microsoft was pursuing with Active Directory, so Samba team just implemented an alternative AD (my main doubt is how the Samba implementation deals with thousands of entries). But anyways the Samba 4 is an enormous step forward for the project and Samba team deserves the appreciation of any open source or linux lover for their gigantic task.
Thanks guys! I am absolutely sure you will continue the good work.
Sunday, November 18. 2012
Debian Multiarch
Debian Multiarch is a new feature for the next version of this popular Linux distribution, in short, it lets a debian box to have binaries of more than one architecture (the most common situation is having amd64 and i386 in the same OS in order to run x64 and x32 applications). As you now I use amd64 testing in my laptop and desktop box and today I have decided to start installing some i386 packages.
Debian announced Multiarch enhancement for the next release (wheeze) in July. I was reluctant to use it at first time cos normally all packages are compiled for both architectures and I do not want them to be duplicated in my box. Nevertheless some months ago I realized that wine had been removed from native amd64 and it had been integrated using i386 and the multiarch feature (since then the amd64 wine package depends on several i386 packages). I suppose that wine has some 32bit tricks which make easier to integrate its i386 implementation. At that moment I just uninstalled it from my system. But some days ago I found that the new version of skype is only distributed in i386 using again the Wheezy multiarch infrastructure. Finally I surrendered and changed my mind under the weight of reality.
The steps to add i386 architecture, skype (4.1 package) and wine in my amd64 laptop were the following:
Add the new architecture i386 to the system.
# dpkg --add-architecture i386 # dpkg --print-architecture amd64 # dpkg --print-foreign-architectures i386
Now amd64 is the native architecture in my system but i386 is also defined as a foreign one. You can remove any foreign architecture using the remove-architecture option for dpkg.
Remove the current (amd64) skype package.
# apt-get remove skype Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: skype 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded. After this operation, 35.5 MB disk space will be freed. Do you want to continue [Y/n]? y (Reading database ... 177126 files and directories currently installed.) Removing skype ... Processing triggers for gnome-menus ... Processing triggers for desktop-file-utils ...
Remove any 32 bit package inside the amd64 architecture. Before the multiarch feature, debian provided several amd64 packages with some important libraries compiled in 32 bits, now they are not needed cos real i386 packages are going to be installed. For that I decided to remove the base, libc6-i386 package, and by consequence all the rest will be auto-removed.
# apt-get remove libc6-i386 Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: ia32-libs ia32-libs-dev ia32-libs-gtk lib32asound2 lib32bz2-1.0 lib32gcc1 lib32ncurses5 lib32nss-mdns lib32stdc++6 lib32tinfo5 lib32v4l-0 lib32z1 libc6-i386 0 upgraded, 0 newly installed, 13 to remove and 0 not upgraded. After this operation, 167 MB disk space will be freed. Do you want to continue [Y/n]? y (Reading database ... 176985 files and directories currently installed.) Removing ia32-libs-gtk ... Removing ia32-libs-dev ... Removing ia32-libs ... Removing lib32asound2 ... Removing lib32bz2-1.0 ... Removing lib32stdc++6 ... Removing lib32gcc1 ... Removing lib32ncurses5 ... Removing lib32nss-mdns ... Removing lib32tinfo5 ... Removing lib32v4l-0 ... Removing lib32z1 ... Removing libc6-i386 ...
Check what packages the skype deb file requires.
# dpkg -I skype-debian_4.1.0.20-1_i386.deb new debian package, version 2.0. size 30688192 bytes: control archive=4629 bytes. 32 bytes, 1 lines conffiles 1079 bytes, 20 lines control 9835 bytes, 137 lines md5sums Package: skype Version: 4.1.0.20-1 Section: non-free/net Priority: extra Architecture: i386 Depends: libasound2 (>= 1.0.16), libc6 (>= 2.3.6-6~), libc6 (>= 2.7), libgcc1 (>= 1:4.1.1), libqt4-dbus (>= 4:4.5.3), libqt4-network (>= 4:4.8.0), libqt4-xml (>= 4:4.5.3), libqtcore4 (>= 4:4.7.0~beta1), libqtgui4 (>= 4:4.8.0), libqtwebkit4 (>= 2.1.0~2011week13), libstdc++6 (>= 4.6), libx11-6, libxext6, libxss1, libxv1, libssl1.0.0 Conflicts: skype-mid, skype-common, skype-bin Replaces: skype-mid, skype-common, skype-bin Installed-Size: 35952 Maintainer: Skype Technologies
Description: Wherever you are, wherever they are Skype keeps you together. Call, see, message and share with others. · It's free to download and join. · Call, instant message and send photos and documents to anyone else on Skype. · And with a webcam you can catch up face-to-face with a video call. · Call mobiles and landlines worldwide at low rates. · Easily text message anywhere in the world. · Get your friends together on a conference call. And that's just the start... Install them using apt-get (the multiarch feature adds to apt-get command the possibility of marking the architecture at install time, if no one is specified the default one is used).
# apt-get install libasound2:i386 libc6:i386 libgcc1:i386 libqt4-dbus:i386 libqt4-network:i386 libqt4-xml:i386 libqtcore4:i386 libqtgui4:i386 libqtwebkit4:i386 libstdc++6:i386 libx11-6:i386 libxss1:i386 libxv1:i386 libssl1.0.0:i386
It seems that the libqtwebkit4 dependency package has a conflicting bug when installing both packages (amd64 and i386) and the installation needs to be forced.
# apt-get install -f -o Dpkg::Options::="--force-overwrite" libqtwebkit4:i386
Install the skype debian multiarch package (now no dependency problem should be found).
# dpkg -i skype-debian_4.1.0.20-1_i386.deb Selecting previously unselected package skype. (Reading database ... 175638 files and directories currently installed.) Unpacking skype (from skype-debian_4.1.0.20-1_i386.deb) ... Setting up skype (4.1.0.20-1) ... Processing triggers for gnome-menus ... Processing triggers for desktop-file-utils ...
Finally I also added wine (as it is distributed by debian, installation adds all the i386 needed packages by its own).
# apt-get install wine
And that's all. I tend to be quite stubborn but, you know, sometimes going with the flow is much easier (nevertheless I will try to be very restrictive with i386 packages).
See you!
Saturday, January 21. 2012
Simple but Full Glassfish HA Using Debian
During the past months I has been working in a project of which environment is mainly Windows (that is the reason of the previous entry about windows mod_proxy_html compilation ). Talking with the customer I realized that Windows 2008 provides a simple and out of the box Load Balancing / High Availability mechanism for TCP/IP client/server infrastructures called Network Load Balancing Services (NLBS). The customer complained about the lack of a similar solution in Linux (remember this customer is pro-windows). During my working life I have always used hardware load balancers in those situations, so I have never investigated how to implement a simple LB/HA solution which only deals with the OS. If you remember I presented an entry about HA in application servers and using an in-memory repository for session management some months ago. Today entry is less ambitious and more practical, I just want to setup a full Glassfish HA setup using only Debian (full means with no single point of failure).
The solution
Looking for information about the problem in the internet, I found lots of LB/HA sample solutions for TCP/IP enviroments in Linux, most of the times they involve two products (see for example this entry):
keepalived. This package provides a VRRP (Virtual Router Redundancy Protocol) implementation stack and a versatile health checking system. So this software can commute a virtual IP between two or more hosts, the health checking system is used to test the servers and decide which one is the master and which one is the backup server. In short keepalived provides High Availability for an IP. In linux there are other more complicated HA techniques to get a full and typical cluster (shared storage, typical virtual -eth0:1 style- IPs,...) but I really do not know much about it.
haproxy. A TCP/HTTP software load balancer that, obviously, adds the Load Balancing part to the solution. In internet examples haproxy is commonly used cos it is generic (it can distribute the load for any TCP/IP protocol: LDAP, HTTP, SMTP, IMAP,...).
But in this entry a particular case is shown, a Java Application Server (Glassfish), and it is clear that there are better techniques for load balancing two Java Servers. For this reason my solution will use two other packages instead of haproxy:
mod_jk. Finally I decided to use the Apache Tomcat Connector as my software load balancer. Glassfish is a very nice piece of software and it has several connectors that can be added to an Apache server to perform this function:
mod_proxy_http: General HTTP reverse proxy module for Apache which supports load balancing.
mod_proxy_ajp: Similar to mod_proxy_http but it manages Apache JServ Protocol version 1.3 (AJP13) to talk with the App Server (Glassfish supports AJP13).
mod_jk: The chosen module also uses AJP13 to connect Apache and the App Server but while mod_proxy_ajp is developed by the httpd group mod_jk is developed by tomcat guys. Usually it is more configurable and better in terms of failure detection.
HTTP Glassfish Load Balancer Plug-in. This plugin is the official Oracle Glassfish load balancer module which is not available in the Open Source Edition. For this reason it was not selected for the demo.
Apache. The famous HTTP web server which is necessary to install mod_jk inside it.
So the solution is quite clear now. First of all keepalived gives a virtual IP which commutes between two Apache/mod_jk boxes (only one httpd server works actively, the other one acts as a backup). The package uses scripts to check if the web server is working fine. The httpd box which has the active IP receives the requests and apache/mod_jk pair redistributes them between two Glassfish servers. So the first layer provides only HA and the second layer LB/HA (take into account that the work in the first layer is lighter, the hard process is performed by the App Server).
My demo environment is the simplest one. Two debian wheezy boxes were virtualized (KVM) inside my laptop: debian1 (192.168.122.21) and debian2 (192.168.122.22). The Linux distribution was installed with the minimal number of packages. Each box will have an Apache/mod_jk and a Glassfish server. Cos the Glassfish 3.1.1 is configured as a cluster debian1 runs the Domain Administrator Server (DAS) too. Replicated session is used (session is sent from one instance to the other when it is changed) but mod_jk uses sticky distribution (the first request is distributed but all the rest from the same source are sent to the same Glassfish server). The keepalived software manages a virtual IP (192.168.122.23) which is assigned to the master host (host with the highest priority). I present a little diagram of my demo solution.
OpenJDK Installation
A Debian solution deserves OpenJDK (the open source JVM):
# apt-get install openjdk-6-jdk
Glassfish version 3.1.1 is going to be used and it is known that there are problems with old versions of JavaSE6. That was the reason to choose wheezy (OpenJDK 6b24) instead of squezze (6b18). JDK was installed in both machines.
Glassfish Installation
The Glassfish installation, although it is not very complicated, requires several steps (I have followed this guide).
I prefer to run glassfish with its own user. So first of all a system user and group were created.
# groupadd glassfish # useradd -c "Glassfish User" -d /opt/glassfish -g glassfish -m -s /bin/bash glassfish # passwd glassfish
This step was done in both machines.
First the linux distribution for Glassfish OpenSource Edition 3.1.1 was downloaded and installed.
$ wget http://download.java.net/glassfish/3.1.1/release/glassfish-3.1.1-unix.sh $ bash glassfish-3.1.1-unix.sh
Please choose to only install the software (do not configure it). I do not like to put installation screens (I am sure that all of you are intelligent enough to install it). I remark only two questions, where to install it (/opt/glassfish/glassfish3.1.1) and what java to use (/usr/lib/jvm/java-6-openjdk-amd64). I completed the installation in both machines.
A domain (domain1) was then created in debian1 (all asadmin commands were launch from /opt/glassfish/glassfish3.1.1/bin).
$ ./asadmin create-domain --savemasterpassword=true --savelogin=true domain1 Enter admin user name [Enter to accept default "admin" / no password]> admin Enter the admin password [Enter to accept default of no password]> Enter the admin password again> Enter the master password [Enter to accept default password "changeit"]> Enter the master password again> Using default port 4848 for Admin. Using default port 8080 for HTTP Instance. Using default port 7676 for JMS. Using default port 3700 for IIOP. Using default port 8181 for HTTP_SSL. Using default port 3820 for IIOP_SSL. Using default port 3920 for IIOP_MUTUALAUTH. Using default port 8686 for JMX_ADMIN. Using default port 6666 for OSGI_SHELL. Using default port 9009 for JAVA_DEBUGGER. Distinguished Name of the self-signed X.509 Server Certificate is: [CN=debian1.demo.kvm,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US] Distinguished Name of the self-signed X.509 Server Certificate is: [CN=debian1.demo.kvm-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US] No domain initializers found, bypassing customization step Domain domain1 created. Domain domain1 admin port is 4848. Domain domain1 admin user is "admin". Login information relevant to admin user name [admin] for this domain [domain1] stored at [/opt/glassfish/.asadminpass] successfully. Make sure that this file remains protected. Information stored in this file will be used by asadmin commands to manage this domain. Command create-domain executed successfully.
In v3 there are no profile types, all domains are equal. After creation domain1 was secured (console uses https instead of plain communication).
$ ./asadmin enable-secure-admin Command enable-secure-admin executed successfully.
As Glassfish is going to use two working instances inside a cluster I usually delete listeners and servers which are not used by the web console.
$ ./asadmin delete-http-listener http-listener-1 Command delete-http-listener executed successfully. $ ./asadmin delete-http-listener http-listener-2 Command delete-http-listener executed successfully. $ ./asadmin delete-virtual-server server Command delete-virtual-server executed successfully.
Now Glassfish v3 uses SSH to communicate with remote instances. So debian2 has to be added to the domain (public key is exchanged for future commands):
$ ./asadmin setup-ssh --generatekey=true debian2.demo.kvm Enter SSH password for glassfish@debian2.demo.kvm> Created directory /opt/glassfish/.ssh /usr/bin/ssh-keygen successfully generated the identification /opt/glassfish/.ssh/id_rsa Copied keyfile /opt/glassfish/.ssh/id_rsa.pub to glassfish@debian2.demo.kvm Successfully connected to glassfish@debian2.demo.kvm using keyfile /opt/glassfish/.ssh/id_rsa Command setup-ssh executed successfully.
And the new node (debian2) was installed:
$ ./asadmin install-node debian2.demo.kvm Created installation zip /opt/glassfish/glassfish3.1.1/bin/glassfish298897142999171245.zip Successfully connected to glassfish@debian2.demo.kvm using keyfile /opt/glassfish/.ssh/id_rsa GlassFish is already installed on debian2.demo.kvm under /opt/glassfish/glassfish3.1.1. Command install-node executed successfully.
Cos glassfish was installed in both machines this command does nothing.
As I said the remote instances are managed via SSH, for this reason there is a new concept called SSH node. These nodes are very different from previous node-agents, they are not a Java daemon at all, they are just remote hosts which run instances of the domain (so they are a concept and not a real daemon or process). Two nodes, called debian1-ssh and debian2-ssh, were created.
$ ./asadmin create-node-ssh --nodehost localhost debian1-ssh Command create-node-ssh executed successfully. $ ./asadmin create-node-ssh --nodehost debian2.demo.kvm debian2-ssh Command create-node-ssh executed successfully.
Then the cluster (cluster1) and the two instances (debian1-gf and debian2-gf) were also created (each instance was registered in the previous nodes and inside the cluster):
$ ./asadmin create-cluster --systemproperties HTTP_SSL_LISTENER_PORT=8181:HTTP_LISTENER_PORT=8080 cluster1 Command create-cluster executed successfully. $ ./asadmin create-instance --cluster cluster1 --node debian1-ssh debian1-gf Command _create-instance-filesystem executed successfully. Port Assignments for server instance debian1-gf: JMX_SYSTEM_CONNECTOR_PORT=18686 JMS_PROVIDER_PORT=17676 ASADMIN_LISTENER_PORT=14848 HTTP_LISTENER_PORT=8080 JAVA_DEBUGGER_PORT=19009 IIOP_SSL_LISTENER_PORT=13820 IIOP_LISTENER_PORT=13700 OSGI_SHELL_TELNET_PORT=16666 HTTP_SSL_LISTENER_PORT=8181 IIOP_SSL_MUTUALAUTH_PORT=13920 The instance, debian1-gf, was created on host localhost Command create-instance executed successfully. $ ./asadmin create-instance --cluster cluster1 --node debian2-ssh debian2-gf Command _create-instance-filesystem executed successfully. Port Assignments for server instance debian2-gf: JMX_SYSTEM_CONNECTOR_PORT=18686 JMS_PROVIDER_PORT=17676 ASADMIN_LISTENER_PORT=14848 HTTP_LISTENER_PORT=8080 JAVA_DEBUGGER_PORT=19009 IIOP_SSL_LISTENER_PORT=13820 IIOP_LISTENER_PORT=13700 OSGI_SHELL_TELNET_PORT=16666 HTTP_SSL_LISTENER_PORT=8181 IIOP_SSL_MUTUALAUTH_PORT=13920 The instance, debian2-gf, was created on host debian2.demo.kvm Command create-instance executed successfully.
Once the instances were running (start-local-instance executed in each node), a new http-listener was registered to listen for the AJP13 protocol (this port will be used later inside mod_jk workers configuration). More details about this configuration in this link:
$ ./asadmin create-http-listener --listenerport 8009 --listeneraddress 0.0.0.0 --defaultvs server --target cluster1 jk-connector Command create-http-listener executed successfully. $ ./asadmin set configs.config.cluster1-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true debian1-gf: configs.config.cluster1-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true debian2-gf: configs.config.cluster1-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true configs.config.cluster1-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true Command set executed successfully.
In order to set the jvmRoute some java and system properties were added (mod_jk uses this property to perform sticky load balancing, in the first request the value of the jvmRoute is appended to the JSESSIONID cookie and the module will use this value to redirect the following requests to the same server).
$ ./asadmin create-jvm-options --target cluster1 "-DjvmRoute=\${AJP_INSTANCE_NAME}" debian1-gf: Created 1 option(s) debian2-gf: Created 1 option(s) Created 1 option(s) Command create-jvm-options executed successfully. $ ./asadmin create-system-properties --target debian1-gf AJP_INSTANCE_NAME=debian1 Command create-system-properties executed successfully. $ ./asadmin create-system-properties --target debian2-gf AJP_INSTANCE_NAME=debian2 Command create-system-properties executed successfully.
In glassfish all instances under a cluster share the same configuration, for this reason different values cannot be set to the same java property in each instance. So the trick is to set a different system property (these props can be defined by instance) and assign the java property referencing the system prop.
Finally the famous clusterjsp was deloyed with high availability enabled (by default glassfish uses session replication). It seems that clusterjsp application is not part of the glassfish 3.1.1 distribution (at least in the linux no multi-language bundle I used), so I created this simple clusterjsp.war (from a previous ear I have from v2 bundle).
$ ./asadmin deploy --availabilityenabled=true --target cluster1 ~/clusterjsp.war Application deployed with name clusterjsp.
At that time both instances could only be managed with *-local-instance commands (launching the commands from its own host). I realized that that was because the master password (password for JKS keystores) was not locally saved (although I swear that I had added savemasterpassword in the domain creation command). To solve that I changed the password in both hosts (I put the same old password cos I just wanted it to be saved in the ~/.asadmintruststore file):
$ ./asadmin change-master-password --savemasterpassword true debian1-ssh Enter the old master password> Enter the new master password> Enter the new master password again> Command change-master-password executed successfully. $ ./asadmin change-master-password --savemasterpassword true debian2-ssh Enter the old master password> Enter the new master password> Enter the new master password again> Command change-master-password executed successfully.
Finally to not use any password in commands at debian2 machine I saved the user and password information as well in this second host (the user and password is saved in ~/.asadminpass file):
$ ./asadmin --host debian1.demo.kvm --port 4848 login Enter admin user name [default: admin]> admin Enter admin password> Login information relevant to admin user name [admin] for host [debian1.demo.kvm] and admin port [4848] stored at [/opt/glassfish/.asadminpass] successfully. Make sure that this file remains protected. Information stored in this file will be used by asadmin commands to manage the associated domain. Command login executed successfully.
Finally I created a simple startup script which needs to be placed in /etc/init.d/glassfish-inst. This script launches a single instance (DAS instance is not needed to be started at boot time, so only the cluster instance will be started in each machine) and it needs asadmin command to not prompt for anything (see the previous step to save the passwords). After the file is correctly placed and the needed variables changed it must be registered in debian.
# chmod 755 /etc/init.d/glassfish-inst # update-rc.d glassfish-inst defaults update-rc.d: using dependency based boot sequencing
So now we have a clustered (session replicated) application. Besides each cluster instance listens for AJP13 protocol in port 8009. Everything is ready for Apache/mod_jk configuration. In this demo I chose Glassfish but the main concepts (maybe changing some pieces) are the same for any application server. And as a final comment for this part, you already know that I personally recommend not to replicate sessions, this process has a big impact if the sessions are too many, too big and/or too volatile (but also be aware that in that case a server lost means a session lost).
Apache/mod_jk Installation
Both packages are part of the main Debian repository, so installing them is just a command like this:
# apt-get install apache2 libapache2-mod-jk
Configuration for JK module is placed in /etc/apache2/mods-available/jk.conf. I just used the debian default but commenting /jk-status and /jk-manager locations (I decided to define them inside the site). This config file points to /etc/libapache2-mod-jk/workers.properties as the workers configuration file. The important lines in the second file are the following:
worker.list=loadbalancer,jk-status worker.debian1.port=8009 worker.debian1.host=debian1.demo.kvm worker.debian1.type=ajp13 worker.debian1.lbfactor=1 worker.debian2.port=8009 worker.debian2.host=debian2.demo.kvm worker.debian2.type=ajp13 worker.debian2.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=debian1,debian2 worker.loadbalancer.sticky_session=1 worker.jk-status.type=status
There are two workers defined, debian1 and debian2 (the name of the worker has to be the same defined in the jvmRoute property), using AJP13 protocol on both virtual hosts with port 8009. Another worker loadbalancer is used to distribute requests between the two previously defined workers. The distribution is configured sticky. Finally the status worker (a page that informs about worker status) is defined with the name jk-status. No special tuning (timeouts and more) was done.
Once mod_jk is configured the default site defined in /etc/apache2/sites-available/default adds two new locations to proxy the clusterjsp app and the status worker:
<Location /jk-status> # Inside Location we can omit the URL in JkMount JkMount jk-status #Order deny,allow #Deny from all #Allow from 127.0.0.1 </Location> # mount clusterjsp JkMount /clusterjsp loadbalancer JkMount /clusterjsp/* loadbalancer
Finally the mod_jk is enabled and apache2 restarted:
# a2enmod jk # /etc/init.d/apache2 restart
These steps have to be done in both hosts (debian1 and debian2). And, at this moment, both Apache servers are balancing the two glassfish clustered instances. As mod_jk detects application server failures the LB/HA of glassfish layer is achieved.
Keepalived Installation
The final package, keepalived, is installed following the debian way:
# apt-get install keepalived
This software has just a little configuration located in /etc/keepalived/keepalived.conf. This file just contains the following:
vrrp_script chk_http_port { # Requires keepalived-1.1.13 script "wget -q -T 1.0 -t 2 --delete-after -O /tmp/test.wget http://localhost:80/index.html" interval 5 # check every 5 seconds weight 2 # add 2 points of prio if OK } vrrp_instance VI_1 { interface eth0 state MASTER # MASTER debian1, BACKUP debian2 virtual_router_id 51 # same id in both hosts priority 101 # 101 on master, 100 on backup virtual_ipaddress { 192.168.122.23 # virtual IP } track_script { chk_http_port } }
A VRRP instance is defined in eth0 interface. This instance manages the IP 192.168.122.23. Host debian1 is declared MASTER with an initial priority of 101, debian2 is the backup instance with a priority of 100. A script chk_http_port is executed in both hosts and it assigns two points more. This way if debian1 Apache fails it only has 101 points while debian2 has 102 (100 + 2 added by the check) and the backup server is transitioned to master (the virtual IP moves from debian1 to debian2). As soon as debian1 Apache runs again (103 points are now assigned to debian1) this host become the master again. If whole debian1 fails (a hardware failure for example), the multicast packets, which are supposed to be sent by the master server, stop being received by the backup server and therefore its status will be risen to master (check VRRP protocol specification in RFC 2338). Take in mind that keepalived does not use typical virtual IPs (eth0:1 and so on), it does HA using VRRP which manages multicast and ARP. VRRP only works inside the same network (both machines have to be inside the same net). With keepalived the first layer is configured in HA (only one apache is working, the other is just awaiting as a backup server).
The checking script is very simple, using wget a test page from the Apache is retrieved (timeout of one second but with two tries). If wget returns 0, the page was got successfully, in case of any other return code, the page was not returned and the check fails. Never test a glassfish page (doing that you are mixing layers and the results can be very weird). Here you have master configuration for debian1 and backup for debian2.
Conclusion
As a conclusion I am going to present a video that shows my demo installation. By default debian1 is the master vrrp server and if clusterjsp page is requested, its apache/mod_jk redirects to any of the glassfish servers (debian1 in the video). Cos the module is configured sticky I can set some attributes in the session and debian1 is always my working glassfish. Now debian1-gf glassfish instance is stopped and clusterjsp is redirected to debian2 (now debian1 apache and debian2 glassfish are working). But then debian1 apache is stopped, debian2 transitions to master state (a tail for /var/log/messages displays keepalived information) and application is still working (now debian2 is doing all the job, apache and glassfish). After restarting debian1 Apache, this host becomes again the master. Finally the virtual machine of debian1 is suddenly stopped, debian2 transitions again to master. Because clusterjsp application was deployed with session replication the session is never lost in the video.
For me it is clear that linux has also a good and simple LB/HA mechanism, keepalived is usually more than enough in typical TCP/IP client/server solutions. With this configuration a virtual IP that commutes between nodes is setup (only HA at the first layer). This layer consists in a software load balancer and, as I commented, usually generic haproxy is used. Nevertheless apache/mod_jk is a much proper solution for AJP13 capable Java Application Servers. The second layer (glassfish in this case) has LB/HA features, cos the load balancer (mod_jk) distributes the load between the two instances at the same time it checks the server status. So the second layer processes requests in parallel. The problem for my customer was that their linux boxes were RedHat6 and keepalived is not distributed by default in the distro (it seems that cluster extra software is needed in order to get this package).
That's why Debian rules!
Comments