Time Machine backup failures on El Capitan with UPS

I have discovered, after a lot of digging, that on El Capitan, at least, the Time Machine will not backup on a desktop or mac mini, if you are connected to a UPS. This appears to be because, since the machine knows it is not a portable, it does not give you the option in Preferences of telling it to back up if on battery power. However, to the system, it appears that you are on battery power when the UPS is the power source. I tried editing the preferences plist file for the time machine, but under El Capitan, those files are locked an you have to disable SIP or System Integrity Package(?) in order to edit those files. Since my system is a server with external users, I don’t like to bounce it up and down to try things out.

Instead, I decided to write a very simple little shell script to ensure that I’d get at least a daily time machine backup. I wrote it in two pieces for testing, but they could easily be combined and also the frequency could be changed. It waits until 6 AM, then checks to see if any time machine backups have been done that day. There should have been 5 or so. If none have been done, then a background job to do the time machine backup is started.

The first script just determines the date of the last time machine backup.

latest_backup


#!/bin/bash
#
# Definition: return the latest backup date
#
tmutil latestbackup | rev | cut -d/ -f1 | cut -d- -f2,3,4 | rev

The second script uses this and determines if it’s not the same day.
If it is not, then it starts the backup job in the background.

backup_if_missed


#!/bin/bash
#
# Description: If latest backup is not current day, do a backup
#
FIRSTV=`latest_backup | tr -d "-" `
SECONDV=`date +%Y%m%d`
DIFF=$((SECONDV-FIRSTV))
if [ $DIFF -ne 0 ]; then
echo "Time machine backup started"
tmutil startbackup &
fi

The last part is to add the following to crontab using sudo crontab -e


0 6 * * * /directory_where_scripts_are/backup_if_missed >> /directory_where_you_put_logs/backups.log

This way, if an update fixes the UPS problem, then I’ll get backups and this script won’t run in addition. But until then, I’ll get a daily time machine backup at 6 AM!

Posted in El Capitan, Observations, OS X, Problems, Recommendations, Source Codes, Tutorials | Tagged , , | Comments Off

Software Development Trends…Troubling IMHO

Background

I’ve been in this business for quite a long time. I started with software during a summer job for the City of Tampa Data Processing Department running IBM unit record equipment including 088 Collators, 057 Interpreters, card sorters and 047 Accounting Machines. These were programmed by using a programming board that consisted of wires connecting “selectors” for various card columns to logic handlers like “and”, “or”, and “not”, then to “action” handlers. Thus in the collator, the wires would be connected such that if a punch was detected in card column 5, then that card would be directed to bin 1, otherwise it would go to bin 2 as an example. I ended up in charge of the “sewer history” accounting, where we calculated the amount of water used by customers and output a new bill for them that included a new charge that helped the city pay for improvements in the sewage system.

After that, we received a very early IBM 360 Model 20! Wow! This was a big change. We read in our compilers from cards, then the program to be compiled, which produced a new binary deck. That was the read in after the loader program was read in from cards, and then a “library” binary deck was loaded. This produced another binary deck that was your executable program that had all library references resolved. This was then read in behind a “job” card. All of this was done in RPG, or Report Program Generator.
Some data was now kept on magnetic tape and could create reports much faster.

After that I got interested in scientific computing and learned Fortran. Scientific programming was mostly spent figuring out how to get the maximum performance out of the hardware on hand and how to create algorithms that would be stable. Since you could only get one run per day, you thought through what the results of the previous run meant, what changes were needed, and what might happen as a result. There was a lot of creating tests that could be analytically verified.

We moved from the IBM 601 drum based machine, to the IBM 709, to the IBM 7090. Then, while working on my masters degree, I got to use an IBM 1401, which used actual “core” memory. My minimization routine for fitting a theoretical calculation to some actual data, ran each night for over four hours for about 4 weeks. It saved its’ results each evening so it could be restarted the next night.

After that I moved up to IBM 360 model 65 machines at the University of Nebraska, where I benchmarked that calculation and it ran for about 6 hours. Later at Louisiana State University, where I was a visiting assistant professor, I ran it on the 360 model 75 for about 4 hours. During this time, I was working on a problem in Atomic Theory and had to really work to get results in any reasonable time by storing intermediate calculations of atomic coupling coefficients and reading them in each time I ran a new calculation. Those coefficients that had not been calculated could be calculated in part by expanding upon earlier coefficients and then storing them for later calculations.

Finally, moving from the academic world to the outside research world, I started as a Computational Physicist at the Oak Ridge National Laboratory, where I worked on simulating Tokomak plasma stability. This was done initially on an IBM 360 model 91, and then later we connected via very slow modems to the Livermore site where we ran our codes on CDC 6400′s then CDC 6600′s. Later, we were able to run on the very early Cray-1, the Cray-XMP, and the Cray-2. For each of these, our calculations were so involved and took so much time that we had to look at the architecture for the machines and then figure out how to fit portions of the code into high speed memory with the other into low speed memory and swap them in and out. With the Cray’s, we spent a lot of time figuring out the best calculational techniques that would use the hardware best. This included such things as noticing that the Cray-1 could get data from 8 memory locations at the same time, IF they are located on separate memory planes. The Cray-1 stored data so that the 1st item was on plane 1, the second on plane 2, etc. Thus by incrementing your DO loops by 8 instead of 1, and in each loop, referencing items 1,2,3…8, then 9,10,…16, you could speed up the calculations by a factor of 8.
This kind of understanding of the hardware architecture and how compilers referenced data, enabled us to approach the theoretical limits of the Cray-1 hardware, which was about 140 MFLOPS, or 140 million floating point operations per second. This doesn’t seem like a lot now, since your laptop is probably 10,000 times faster that that now.

What this does show is that there was a lot of focus on understanding and on getting results. A lot of attention had to be given to the effects of the data word length upon the accuracy of the calculations.

Later in moving to producing applications for clients, we moved to Object Oriented approaches like Objective-C and Java, which gave us much better quality and stability IF we designed carefully and developed with an eye on maintenance and quality. Developing on a modern laptop with very nice IDEs, immensely improves productivity, but at the risk of quick and dirty fixes without concern for their side-effects. I’ve learned to write a LOT of tests, often using Test Driven Development so that I end up with a very large regression suite and to focus on creating solutions wherein as much of the application as possible has been externalized, e.g. flow becomes XML state models, logic becomes XML rules, persistence information becomes XML or JSON data, and the major portions of the applications are robust web-services. Presentation is also externalized using CSS, XSL, XML, JSON, and HTML, with navigation handled using the state models.

Changes in attitudes

What I see now is an ever increasing number of ways to create newer and slicker buttons and drop downs with lots of customized pages, with very little attention to performance or quality or even content. Every month there seems to be a new jQuery or other javascript framework to handle some trivial example and show how slick the sliders work. Each time that I try to adapt these amazing new “frameworks” to a real life problem, I find that they are so rigid and so fragile, that I would never want to release them to a client as part of a solution. The focus seems to be on appearance and upon quick and easy to whip up some trivial example. All of the information about each new “improvement” is some kind of internal knowledge that is not well documented and only available to those “in the know”.

Finding solutions on the web

In addition, those kind souls that answer questions via the web forums, seem to leave out little details like putting a date on their answer, specifying the version number of the OS, the libraries, the JVMs, or anything else. Some sites are better than others and some mentors better also, but the general quality of answers seem to be quick “Why would you do it that way? I’d use blizt-fraz instead”, or “Just bazz-fazz on the croten-frotz and it will fix everything”, or “Do this…X”, of course when you do X, it doesn’t work, because you carefully said you were using a Mac, but they give you a solution for Windooze, and don’t mention that fact.

Recommendations

Mentors, and you are appreciated for trying, please read the question carefully. Ask for more information about the environment and situation to answer professionally. Look at what is involved in your answer. If it requires specific versions of software or libraries, specify them. Often the person asking the question is on a project, with restrictions about their environment so suggestions to just change to X aren’t real helpful. For those asking questions, give a date, your OS version, your JVM version, if appropriate, and other environment data so that the answer can be useful to you. State your question in a clear manner that isolates your specific problem and be professional. Our profession is hard. It borders on magic in some sense. You give a very long incantation ( in software ), that you hope will produce a beautiful user experience, but one minor little error can lead to an absolute catastrophe!

Posted in Architecture, Development, Hardware, Observations, Physics, Problems, Recommendations | Comments Off

Wigner 3J in FORTRAN

I’ve been trying to reincarnate an old FORTRAN Atomic Physics code from many years ago that I had kept around but had misplaced one subroutine called ANGL. It calculated Wigner 3J coefficients and I had spent some time figuring out how to do that in the dark ages.

With the modern Internet, I thought it would be a simple search and I’d find someone who had created a standard version of this. I’d then download it, adapt its argument list to match the requirements in my code and be done. Silly me!

It appears that while the definitions are available and there are Wolfram calculators and javascript calculators on line, there isn’t a FORTRAN version that I could find that would compile and work. I found a couple in C++, but they didn’t compile and work on my MacBook Pro, either.

It seems like it is some sort of dark secret how to actually calculate a given 3J symbol for use in a FORTRAN code. When looking back through the literature on line, each author seems to delight is switching notations and giving you definitions in terms of other definitions. Wikipedia isn’t much help here!

So, repeating the quest I did so many years ago, I worked out a version that works for my purposes and might be helpful as a starting point for some other poor slob in a similar boat. I’ll include the definition by Racah of the Wigner 3J symbols, and then show how they are implemented in the FORTRAN subroutine ANGL. Of course, it should be a simple task to take what I’m presenting here and port it into C, C++, Objective-C, or Java.

We start with the Racah definition of the Wigner 3J symbol.

Wigner 3J Racah definition
In our calculations this is done by calculating tmin and tax as

t1 = j2 - j3 - m1
t2 = j1 - j3 + m2
t3 = j1 + j2 - j3
t4 = j1 - m1
t5 = j2 + m2
tmin = min( 0, min( t1, min(t2, min(t3, min(t4, t5)))))
tmax = max( t1, max(t2, max(t3, max(t4, t5))))

The codes below are written in FORTRAN 66 maybe or FORTRAN 77. That’s what I was going back to to start. I’m sure someone can convert these to more modern FORTRAN, but I liked the simplicity and wanted to show every step, not be clever!

The file ANGL.f contains subroutine ANGL. It does not check anything but conditions about all arguments being integer or half-integer. The accompaning subroutine QANG, which would be in the file QANG.f, does the checking and is the actual call to ANGL, determining those which are automatically zero and also has the framework for storing and retrieving previously calculated symbols. This will be updated in a later post. In addition, I’ve included the file Wigner3JMain.f to show how QANG is called, along with a shell script, run3J.sh which compiles and runs the software.

File ANGL.f

      REAL*8 FUNCTION ANGL(j1, j2, j3, m1, m2, m3)                      ANGL0010
      IMPLICIT REAL*8(A-H,O-Z)
      REAL*8 wigner, delta
      COMMON numFactorials, ifact(10)
      INTEGER t1, t2, t3, t4, t5, tmin, tmax
9999  FORMAT(I5)
      ANGL = -999.9
      if (j1+j2+j3+1 > 10 ) GOTO 100
      ANGL = 0.0
      if( 2*j1 .NE. floor(2.0*j1) ) GOTO 100
      if( 2*j2 .NE. floor(2.0*j2) ) GOTO 100
      if( 2*j3 .NE. floor(2.0*j3) ) GOTO 100
      if( 2*m1 .NE. floor(2.0*m1) ) GOTO 100
      if( 2*m2 .NE. floor(2.0*m2) ) GOTO 100
      if( 2*m3 .NE. floor(2.0*m3) ) GOTO 100
   10 CONTINUE
      t1 = j2 - j3 - m1
      t2 = j1 - j3 + m2
      t3 = j1 + j2 - j3
      t4 = j1 - m1
      t5 = j2 + m2
      tmin = min( 0,  min( t1, min(t2, min(t3, min(t4, t5)))))
      tmax = max( t1, max(t2, max(t3, max(t4, t5))))
      wigner = 0.0
C     Calculate Sum over I of (-1)^I/x
C     Where x is I!(j3-j2+I+m1)!(j3-m1+I-m2)!(j1+j2=m3-I)!(j1-I-m1)!(j2-I+m2)!
      DO 9 I = tmin, tmax
      IX = ifactr(I)
      IX = IX * ifactr(j3-j2+I+m1)
      IX = IX * ifactr(j3-j1+I-m2)
      IX = IX * ifactr(j1+j2-j3-I)
      IX = IX * ifactr(j1-I-m1)
      IX = IX * ifactr(j2-I+m2)
C      CALCULATE (-1)**I
      IY = (-1)**I
      wigner = wigner + float(IY)/float(IX)
9     CONTINUE
C     Calculate the triangle_coeff idelta
      delta = float(ifactr(j1+j2-j3)*ifactr(j1-j2+j3)*ifactr(-j1+j2+j3))
      delta = delta/float(ifactr(j1+j2+j3+1))
C     Calculate sign
      I1 = (-1)**(j1-j2-m3)
C     Calculate the factorials in the square root with delta
      IY = ifactr(j1+m1)
      IY = IY * ifactr(j1-m1)
      IY = IY * ifactr(j2+m2)
      IY = IY * ifactr(j2-m2)
      IY = IY * ifactr(j3+m3)
      IY = IY * ifactr(j3-m3)
      ANGL = I1 *  wigner*sqrt(FLOAT(IY)*delta)
1000  FORMAT(1H1,46HAll arguments must be integer or half-integer.6(I5))
 100  RETURN
      END
      INTEGER FUNCTION ifactr(K)
      INTEGER numFactorials
      COMMON numFactorials, ifact(10)
      ifactr = 1;
      if( K == 0 ) GOTO 10
      ifactr = ifact(K)
10    CONTINUE
      RETURN
      END
      INTEGER FUNCTION ifactorial( K )
      J = 1;
      IF ( K .EQ. 0 ) GO TO 11
      DO 10 I=1,K
      J = J*I
  10  CONTINUE
  11  ifactorial = J
      return
      END

File QANG.f

      FUNCTION QANG(J1,J2,J3,M1,M2,M3)                                  QANG0020
      IMPLICIT REAL*8(A-H,O-Z)                                          QANG0040
      REAL*8 ANGL
      INTEGER*2 JTABLE(700,3)                                           QANG0060
      DIMENSION TABLE(74)                                               QANG0080
      DATA NUM/503/                                                     QANG0100
      DATA IFLAG/1/                                                     QANG0120
      DATA ITEST/1/
      IF( ITEST .NE. 1 ) GO TO 11
      DO 9 I = 1, 700
      DO 9 J = 1,3
      JTABLE(I,J) = -1
 9    CONTINUE
 11   CONTINUE
      IF(IFLAG.NE.0)GO TO 989                                           QANG0480
C...  INITIALATION AND SORTED TABLE AND ITABLE GO HERE                  QANG0500
      READ(10)((JTABLE(I,J),J=1,3),I=1,700)                             QANG0520
      READ(10)(TABLE(I),I=1,74)                                         QANG0540
  989 IFLAG=1                                                           QANG0560
      QANG = 0.D0
      IF( IABS(M1) .GT. J1 ) GOTO 973
      IF( IABS(M2) .GT. J2 ) GOTO 973
      IF( IABS(M3) .GT. J3 ) GOTO 973
      IF( IABS(J1-J2) .GT. J3 ) GOTO 973
      IF( J3 .GT. J1+J2 ) GOTO 973
      IF( M1+M2+M3 .NE. 0) GOTO 973
      GOTO 1073
      INDEX=J1*100000+J2*10000+J3*1000+M1*100+M2*10+M3                  QANG0700
      IQUOT=INDEX/NUM                                                   QANG0720
      IPROB=INDEX-IQUOT*NUM+1                                           QANG0740
C      WRITE(6,9999) INDEX
C      WRITE(6,9999) IPROB
      IF(JTABLE(IPROB,1).EQ.-1)GO TO 999                                QANG0760
  100 CONTINUE                                                          QANG0780
      IF(JTABLE(IPROB,1).NE.IQUOT)GO TO 200                             QANG0800
      QANG=TABLE(JTABLE(IPROB,2))                                       QANG0820
      GO TO 973                                                         QANG0860
  200 IF(JTABLE(IPROB,3).EQ.-1)GO TO 999                                QANG0880
      IPROB=JTABLE(IPROB,3)                                             QANG0900
      GO TO 100                                                         QANG0920
  999 QANG=0.D0                                                         QANG0940
 1073 CONTINUE                                                          QANG1000
      QANG=ANGL(J1,J2,J3,M1,M2,M3)                                      QANG1020
C      TABLE(IPROB,1) = IQUOT
C      TABLE(IPROB,2) = QANG
  973 CONTINUE                                                          QANG1040
9999  FORMAT(1H1,I10)
      RETURN                                                            QANG1060
      END                                                               QANG1080

File Wigner3JMain.f

C      PROGRAM WIGNER
C..... MAIN PROGRAM FOR TESTING AND RUNNING QANG
      REAL*8 X, QANG, ANGL
      INTEGER numFactorials

      COMMON numFactorials, ifact(10)
C
      numFactorials = 10
C
      DO 8, I=1,numFactorials
      ifact(I) = ifactorial(I)
 8    CONTINUE
      WRITE(6,9999) ifactr(0), ifactr(1), ifactr(2),ifactr(10)
C       GOTO 101
9999   FORMAT(3I5I30)
       I=2
       J=2
       K=0
       II=0
       JJ=0
       KK=0
       X = ANGL(I,J,K,II,JJ,KK)
C      FOR (2,2,0 / 0,0,0 ) = sqrt(1/5)
	    WRITE(6,1001)I,J,K,II,JJ,KK,X
       I=1
       J=1
       K=0
       X = ANGL(I,J,K,II,JJ,KK)
	    WRITE(6,1001)I,J,K,II,JJ,KK,X
       X = QANG(I,J,K,II,JJ,KK)
	    WRITE(6,1001)I,J,K,II,JJ,KK,X
C       GO TO 101
       Write(6,1000)
1000   FORMAT(1H1,"Starting to calculate some 3J symbols\n")
	   DO 100 I = 0,2
	   DO 100 J = 0,2
	   DO 100 K = 0,2
	   DO 100 II = 0,2
	   DO 100 JJ = 0,2
	   DO 100 KK = 0,2
	   X = QANG(I,J,K,II,JJ,KK)
      IF( X .EQ. 0.0 ) GOTO 100
	   WRITE(6,1001)I,J,K,II,JJ,KK,X
1001   FORMAT(1H1,I3,1X,I3,1X,I3,1X,I3,1X,I3,1X,I3,1X,F10.3)
100    CONTINUE
101    CONTINUE
	    WRITE(6,300)
300    FORMAT(1H1,"ALL DONE!\n")
       STOP
       END

The shell script to run this on Mac OS X 10.10.3 using the gcc-4.9.bin after installing XCode 6.3 and the command line tools ( Mac OS X has made it difficult to use the underlying Unix functionality by moving things to unconventional places. Unless you install XCode and the command line tools, you are missing most of the Unix command line features. )

File run3j.sh

gfortran -v
gfortran -o Wigner -lcrt1.o -ffpe-trap='underflow' Wigner3JMain.f QANG.f ANGL.f
Wigner

The output for running the above shell, if everything works right and all the fortran guts are installed correctly, etc. is:

19:48:40 [woo:~/Development … e/fortran/3J] > run3J.sh
Using built-in specs.
COLLECT_GCC=gfortran
COLLECT_LTO_WRAPPER=/usr/local/libexec/gcc/x86_64-apple-darwin13.0.0/4.9.0/lto-wrapper
Target: x86_64-apple-darwin13.0.0
Configured with: ../gcc-4.9-20131215/configure --enable-languages=c++,fortran
Thread model: posix
gcc version 4.9.0 20131215 (experimental) (GCC) 
    1    1    2                       3628800
1  2   2   0   0   0   0      0.447
1  1   1   0   0   0   0     -0.577
1  1   1   0   0   0   0     -0.577
1Starting to calculate some 3J symbols\n
1  0   0   0   0   0   0      1.000
1  0   1   1   0   0   0     -0.577
1  0   2   2   0   0   0      0.447
1  1   0   1   0   0   0     -0.577
1  1   1   0   0   0   0     -0.577
1  1   1   2   0   0   0      0.365
1  1   2   1   0   0   0      0.365
1  2   0   2   0   0   0      0.447
1  2   1   1   0   0   0      0.365
1  2   2   0   0   0   0      0.447

I’ll put these files into a download directory soon and arrange a link to them. Good luck!

Posted in Interests, Mathematics, Physics, Source Codes | Comments Off

Lorem ipsum on OS X

Finally figured out how to implement Lorem Ipsum on Mac OS X 10.10.2.
This is handy when developing and wanting to insert a longish string of text to check the word wrap or formatting on some page. It’s nice because the text doesn’t distract the user by having them try to read and criticize grammar or content.

I created it on my Mac PowerBook running 10.10.2 version of OS X by opening the Automator application( provided in the OS). Then click New, select “Service”, and in the small search window, type run. From the choices, choose ‘Run AppleScript.’ Drag that icon to the workspace to the right. Copy what is below and replace the contents in the workspace with the copied script. Save the script with some name, such as “Lorem”. Then open any text document, select some small amount of text where you want the paragraph. Then under the File menu for the text application you are using, select ‘Lorem’. Viola, you should see your piece of text replaced by the Lorem Ipsum text!

on run {input, parameters}

set new_text to "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum." & input

return new_text
end run

Posted in Development, Problems, Recommendations, Software | Tagged , , , , , , , | Comments Off

Creating a speed test to and from your server

We can easily find our upload and download speeds between our laptops and some network hub using such tools as: http://speedtest.comcast.net/

The real question, if you have users connecting to your personal server, is what are the upload and download speeds between their laptops and your server. How do we measure this? I looked around and it appeared that everyone says you need special tools for this. I’m not sure, but you can get a pretty good idea using the command line tools and having ssh set up between your laptop and the server. Here’s how.

Find a file of about 1-2 megabytes on your laptop. On a Mac, get the info and copy the exact number of bytes. Paste it into the following shell script:


# UNIX timestamp concatenated with nanoseconds
T="$(date +%s)"

# Do some work here
BYTES=1499244
sftp -b upload.cmds userId@yourserver

# Time interval in nanoseconds
T="$(($(date +%s)-T))"
# Seconds
S="$((T/1000000000))"
# Milliseconds
M="$((T/1000000))"

echo "Time : ${T} to upload $BYTES bytes"
printf "Pretty format: %02d:%02d:%02d:%02d.%03d\n" "$((S/86400))" "$((S/3600%24))" "$((S/60%60))" "$((S%60))" "${M}"
echo "Upload speed: $((BYTES/T)) bytes/sec"

Create the following upload.cmds file:

cd tmp
put pathToDirectoryWhereFileIsLocated/xerces-1.2.3.jar
quit

Now, make the uploadTest.sh file executable, e.g. chmod a+x uploadTest.sh

When it executes, you should see:


08:00:50 ⌁ [woo:~/bin] 31s > uploadTest.sh
sftp> cd tmp
sftp> put pathToLocationOfFile/xerces-1.2.3.jar
Uploading pathToLocationOfFile/xerces-1.2.3.jar to pathToTempFileYourDirectory/tmp/xerces-1.2.3.jar
sftp> quit
Time : 6 to upload 1499244 bytes
Pretty format: 00:00:00:00.000
Upload speed: 249874
All Done

Posted in Observations, Problems, Recommendations, Software | Tagged , , , | Comments Off

OS X Yosemite Beta

Just downloaded and installed Yosemite beta-2. Be investigating it for a few weeks, but my first impression is that I don’t like the direction that Apple is moving with the visual look and feel.
Apple had high resolution and great icons with depth and it really made Windooze and Linux look amateurish. Now, everything has moved to that flat, pale, unimpressive look. So unprofessional looking and blah! In addition, the move keeps going to dumb down the tools and development tools. As an old time command line developer, I can rename dozens of files or change their permissions, etc. with a simple piped set of commands. Now, more and more of the command line tools get removed or are unavailable to users who wish them. Everything becomes a multiple set of clicks. To change a dozen files might take 36-48 separate clicks!
At least it would be nice if one could set “expert” somewhere in the preferences panel and get the underlying BSD Unix commands exactly as they have always been. It would be nice if a lot of that stuff was also kept in the same places they have historically been.
My opinion only, your mileage might vary.

Posted in Observations | Tagged , , , , | Comments Off

Carbonite Fail

Got to thinking about the fact that I have my computers backing up to a Time Capsule, but that capsule is located 15 feet from my computer. A fire in just the one room would wipe out everything I’ve accumulated during years of software development. While I also use Dropbox, it isn’t really an off-site backup solution as I have to have the files located in Dropbox. I’ve found that that is really not useful for dozens of application development projects, plus as you compile or edit files, things slow down while the synchronization occurs.

I decided to look into Carbonite. For a “normal” user, who only edits documents, does email, uses spreadsheets, iTunes, iPhoto, it’s fine. However, the power user, who has installed TeX or uses ports to install what might be called UNIX software that goes into /usr/local or /opt/…,
it provides no help. You web site isn’t located in your user area, but under some system areas. These CAN’T be backed up using Carbonite. Oh, you could have a sync process that nightly copies them to your user area so they would get backed up, but that’s a bit ridiculous.

It looks like I’m going to have to fall back to an old tried and true method of setting up a computer and drive at an off-site location, like my companies office, and using rsync to periodically copy stuff to it. That way I can have a clone of my computer kept in near time sync.

Posted in Problems, Recommendations, Software, Uncategorized | Tagged , , , , , | Comments Off

Simple Test Harness for GUI Applications

I have been working for some time to add some extensions to an open source java application called FreeMind. While I wrote many class level and low level JUnit tests as a regression suite, I still wanted to automate the testing of new features at the user level, which is GUI based, of course. This means I have to start up the application, wait for the window to appear, then manage the clicking on various menu items, get the responses, and then validate that they are what would be expected.

Another open source project for GUI level testing called abbot is available and it provides methods to handle clicks and so forth. However, as far as I could determine, it didn’t work very well if the software is tested on different monitors with different resolutions as it depended upon where the buttons were on the screen. Also, the FreeMind application saves a user preferences which includes where the screen shows up and its size. Now, if you test it on your computer, setting all of the click positions, and someone else ( my associate who helps me with testing ), their position and window size could be different. While there is a companion program for abbot called costello ( of course ) which can record clicks and create a script for abbott, I could not get it working with FreeMind because of the manner that FreeMind starts up. It does some strange things to handle its startup so that it can run as an applet or a regular application, plus lots of creating of classes from xml files, etc.

So…., what to do?

I looked around, remembered some things I had done in the past with state machines, reading the state model information from xml files, etc. and decided to create a very simple state machine-like Test Harness. Now, I want to start the application, then in a series of JUnit tests, run various test methods to test user-level interactions. In JUnit tests the setUp method in each test class is run before each test method. This allows you to reset things so that each test is not affected by the previous test. If you want to have tests with several things in sequence, then you make your test method do that. Or, you can use a static boolean set at the end of one method to tell the next method not to reset everything. Test methods seem to run in alphabetic order, or at least in the order they are in the test case and the way we do our Eclipse setup, the methods are sorted alphabetically, so it’s the same thing.

In addition, from previous work, I wanted to use the same xml test files that are executed to run the test methods to provide documentation for our test plan, which we maintain in LateX, and for web pages describing each one. Thus before releasing the application, we run all tests again, the final test case of which actually updates both the LaTeX test plan and the web pages, so that they are completely consistent with the actual tests. More on this later.

First I wrote up a test script in xml that did what I think I needed to accomplish. I needed a test name and description, then a sequence of “actions” each of which had a name like “clickButton”, and a value which told me which button to press. Here is our first test script, which we called, Test001.xml.

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE test SYSTEM "tests.dtd" >
   <test>
      <name>Test001</name>
      <description>Checkout a map, delete random node, commit and verify</description>
     <action>
            <name>clickMenuItem</name>
            <value>Checkout...</value>
      </action>
     <action>
            <name>waitForButtonOnPanelToAppear</name>
            <value>0</value>
      </action>      
     <action>
            <name>clickButton</name>
            <value>Yes</value>
      </action>      
      <action>
            <name>waitForMap</name>
            <value>TeleICU</value>
      </action>
     <action>
            <name>selectNode</name>
            <value>randomLeaf</value>
      </action>
     <action>
            <name>clickFreeMindMenuItem</name>
            <value>Remove Node</value>
      </action>
     <action>
            <name>waitForButtonOnPanelToAppear</name>
            <value>0</value>
       </action>      
     <action>
             <name>clickButton</name>
            <value>Yes</value>
       </action>           
     <action>
            <name>commitMap</name>
            <value>comment</value>
      </action>
     <action>
            <name>pause</name>
            <value>2</value>
      </action>
     <action>
            <name>validate</name>
            <value>CheckDatabaseForChangedNodes</value>
      </action>
   </test>

Note that we did create a dtd so that we would get warnings if we didn’t make the xml file correct.
That file is: tests.dtd

<!ELEMENT test (name , description , action+ ) >

<!ELEMENT name (#PCDATA)>

<!ELEMENT description (#PCDATA)>

<!ELEMENT action (name , value) >

<!ELEMENT value (#PCDATA)>

So, as you can see in the sample test, it has a name and a description, followed by the sequence of actions. For our extension, we wanted to “check out” a map from an archive. That meant clicking on the Menu Item called “Checkout…”, waiting for the panel to come up with the list of archived maps, selecting the 0th one, in this case, clicking the “OK” button on that panel, and waiting for the map to come up. Then, select a random node in the map and delete it by clicking on the FreeMind menu item ( now ours ) for “Remove Node”, clicking on the “confirm” button, then “committing” the map, i.e. saving it as revised by clicking on our “Commit…” menu item, clicking on the button to really commit it, then waiting for the map to reappear as revised, and then verifying that the archive store was altered as we expect it to be.

We chose to use a Digester approach( because I knew how to use them ) and here is our TestModule class:

import org.apache.commons.digester3.binder.AbstractRulesModule;

public class TestModule extends AbstractRulesModule
{

   @Override
   protected void configure()
   {

      // Register a ObjectCreatRule on matching pattern "test". Later on, in the parsing phase,
      // when encounters an  element, the digester will fire this rule to create an Action object.
      // Also register a SetPropertiesRule on the same pattern. Later on, in the parsing phase,
      // the digester will fire this rule to set properties of the Test object 
      // with the attribute values of the  element
      // For the setProperties() to work this way, a property name must be the same as the attribute name.
      forPattern( "test" ).createObject()
         .ofType( "testHarness.GuiTest" ).then()
         .setProperties();

      // Register a BeanPropertySetterRule on matching pattern "test/name", to
      // set the property of the Test object named 'name' with the content of the 
      // element.  Also do this for the description property.
      forPattern( "test/name" ).setBeanProperty();
      forPattern( "test/description" ).setBeanProperty();

      // Register an ObjectCreateRule to create an Action object when the action tag is encountered.
      // use the addAction method in the Test object to add the Action to the list of actions.
      forPattern( "test/action" ).createObject()
         .ofType( "testHarness.Action" ).then()
         .setProperties().then().setNext( "addAction" );

      // Register a BeanPropertySetterRule on matching pattern "test/action/name", to
      // set the property of the Action object named 'name' with the content of the 
      // element.  Also do this for the value property.

      forPattern( "test/action/name" ).setBeanProperty();
      forPattern( "test/action/value" ).setBeanProperty();

   }
}

Now, this is invoked by the Test Creator:

import java.io.IOException;
import java.io.InputStream;

import org.apache.commons.digester3.Digester;
import org.apache.commons.digester3.binder.DigesterLoader;
import org.xml.sax.SAXException;

public class TestCreator
{

   /**
    * Creates a Family object (and Address, Member objects contained by it)
    * based on XML data.
    * 
    * @param source
    *           - name of the XML file
    * 
    * @throws SAXException
    * @throws IOException
    */
   public static GuiTest createTest( String source ) throws SAXException,
      IOException
   {
      GuiTest result = null;
      InputStream inputStream = TestModule.class.getClassLoader()
         .getResourceAsStream( source );

      DigesterLoader digesterLoader = DigesterLoader
         .newLoader( new TestModule() );
      digesterLoader.setValidating( true );
      Digester digester = digesterLoader.newDigester();
      MyErrorHandler eh = new MyErrorHandler();
      digester.setErrorHandler( eh );
      result = digester.parse( inputStream );

      return result;
   }
}

Note that we added an errorHandler, MyErrorHandler, but this is not absolutely necessary. Also, the setValidating isn’t necessary if you don’t want it to use the dtd file for validation.

The class GuiTest referred to in the TestModule is:

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class GuiTest
{
   private String       name;
   private String       description;
   private List actions = new ArrayList();

   public void addAction( Action action )
   {
      actions.add( action );
   }

   public List getActions()
   {
      return Collections.unmodifiableList( actions );
   }

   public String getDescription()
   {
      return description;
   }

   public String getName()
   {
      return name;
   }

   public void setDescription( String description )
   {
      this.description = description;
   }

   public void setName( String name )
   {
      this.name = name;
   }

   @Override
   public String toString()
   {
      return name + ": " + description;
   }
}

And the class Action is:

public class Action
{
   private String name;
   private String value;

   public String getName()
   {
      return name;
   }

   public String getValue()
   {
      return value;
   }

   public void setName( String name )
   {
      this.name = name;
   }

   public void setValue( String value )
   {
      this.value = value;
   }

}

Now, we wrote a TestHarnessTester to be sure it parsed correctly and had the methods we needed.

import java.io.InputStream;
import java.util.List;

import junit.framework.Assert;

import org.junit.BeforeClass;
import org.junit.Test;

public class TestHarnessTest
{
   private static GuiTest test  = null;
   private int            COUNT = 11;

   @Test
   public void testAction()
   {

      List actions = test.getActions();
      Assert.assertEquals( "size not correct", COUNT, actions.size() );
      Action action = actions.get( 0 );
   }

   @Test
   public void testActionMethod()
   {
      List actions = test.getActions();
      Assert.assertEquals( "size not correct", COUNT, actions.size() );
      Action action = actions.get( 0 );
      Assert.assertEquals( "name wrong", "clickMenuItem", action.getName() );
      Assert.assertEquals( "value wrong", "Checkout...", action.getValue() );
   }

   @Test
   public void testActions()
   {
      List actions = test.getActions();
      Assert.assertEquals( "size not correct", COUNT, actions.size() );
      Action action = actions.get( 1 );
      Assert.assertNotNull( "should have method", action );
      Assert.assertEquals( "name wrong", "waitForButtonOnPanelToAppear", action.getName() );
      Assert.assertEquals( "value wrong", "0", action.getValue() );
      action = actions.get( 2 );
      Assert.assertEquals( "name wrong", "clickButton", action.getName() );
      Assert.assertEquals( "value wrong", "Yes", action.getValue() );
   }

   @Test
   public void testTest()
   {
      Assert.assertEquals( "Incorrect test name", "Test001", test.getName() );
      Assert
         .assertNotNull( "Incorrect test description", test.getDescription() );
      System.out.println( "Test: " + test.getName() + " - "
         + test.getDescription() );
   }

   @BeforeClass
   public static void setup()
   {
      String resource = "Test001.xml";
      String source = "testHarness/" + resource;
      System.out.println( TestHarnessTest.class.getClassLoader().getResource(
         source ) );
      InputStream inputStream = FamilyModule.class.getClassLoader()
         .getResourceAsStream( source );
      if ( inputStream == null )
      {
         System.out.println( "InputStream not located for " + source );
         System.exit( 0 );
      }
      try
      {
         test = TestCreator.createTest( source );
      }
      catch ( Exception e )
      {
         System.out.println( "Exception " + e );
         System.exit( 1 );
      }
   }
}

Now, we put our Test xml files in a folder “testScripts” located under the “src” folder in our test package in Eclipse. This ran and completed all tests sucessfully.

This harness can be used for a lot of testing purposes. Here I show a simple TestCase that will run the script Test001.xml.

import junit.extensions.abbot.TestHelper;

import org.junit.Test;

public class GUITest001 extends ArchiverGUITestBase
{
   public GUITest001( String name )
   {
      super( name );
      CLEAN_DB = "../archiverDoc/sqlScripts/ThreeMaps.sql";
      MAPNAME = "icu_tele.mm";
   }

   @Test
   public void test001()
   {
      loadTestXML( "testScripts/Test001.xml" );
      runTestScriptAndReloadDataBase();
   }

   @Test
   public void test002()
   {
      loadTestXML( "testScripts/Test002.xml" );
      runTestScriptAndReloadDataBase();
   }

   public static void main( String[] args )
   {
      TestHelper.runTests( args, GUITest001.class );
   }

}

This test shows two tests, one not given here, but we load the test script, then run it and optionally reset the database or not for the next test.

Here is the class ArchiverGUITestBase:

import java.awt.AWTException;
import java.awt.Component;
import java.awt.Container;
import java.awt.Point;
import java.awt.Robot;
import java.awt.event.InputEvent;
import java.awt.event.WindowEvent;
import java.awt.event.WindowFocusListener;
import java.io.File;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;

import javax.swing.ImageIcon;
import javax.swing.JButton;
import javax.swing.JDialog;
import javax.swing.JLabel;
import javax.swing.JMenu;
import javax.swing.JMenuItem;
import javax.swing.JPanel;
import javax.swing.JTextField;
import javax.swing.RootPaneContainer;
import javax.swing.SwingUtilities;
import javax.swing.UIManager;

import junit.extensions.abbot.ComponentTestFixture;
import junit.extensions.abbot.TestHelper;
import abbot.finder.Matcher;
import abbot.tester.ComponentTester;

import testHarness.Action;
import testHarness.GuiTest;
import testHarness.TestCreator;
import testHarness.TestHarnessTest;
import testHarness.TestModule;

import freemind.main.FeedBack;
import freemind.main.FreeMind;
import freemind.main.FreeMindStarter;
import freemind.main.Tools;
import freemind.modes.MindMapNode;
import freemind.modes.ModeController;
import freemind.view.mindmapview.NodeView;

public class ArchiverGUITestBase extends ComponentTestFixture
{

   public static String          CLEAN_DB         =
                                                     "../archiverDoc/sqlScripts/mm_test_script.sql";

   public ComponentTester        tester;

   public String                 gotClick;

   public static final int       DELAY            = 1;

   public static FreeMindStarter fms              = new FreeMindStarter();

   public static FreeMind        fmm              = null;

   public static boolean         loadTest         = true;

   public static String          lastTestXML      = "";

   public static String          testXML;

   public static GuiTest         test             = null;

   public static HarnessHelper   harnessHelper    = null;

   public ArchiverGUITestBase()
   {

   }

   public ArchiverGUITestBase( String name )
   {
      super( name );
      init();
   }

   public Point click( int x, int y )
   {
      Point p = new Point( x, y );
      try
      {
         Robot robot = new Robot();
         TestUtils.pause( DELAY );
         robot.mouseMove( x, y );
         TestUtils.pause( DELAY );
         robot.mousePress( InputEvent.BUTTON1_MASK );
         robot.mouseRelease( InputEvent.BUTTON1_MASK );
      }
      catch ( AWTException e )
      {
         // TODO Auto-generated catch block
         e.printStackTrace();
      }
      TestUtils.pause( DELAY );
      return p;
   }

   public void init()
   {
      Properties defaultProperties = fms.readDefaultPreferences();
      File autoPropertiesFile =
         new File( "/Users/" + System.getProperty( "user.name" )
            + "/.freemind/auto.properties" );
      String[] args = {};
      setUserCredentials();
      if ( fmm == null )
      {
         fmm =
            new FreeMind( defaultProperties,
               fms.readUsersPreferences( defaultProperties ),
               autoPropertiesFile );
         final FeedBack feedBack;
         feedBack = new FeedBack()
         {
            int value = 0;

            @Override
            public int getActualValue()
            {
               return value;
            }

            @Override
            public void increase( String messageId )
            {
               progress( getActualValue() + 1, messageId );
            }

            @Override
            public void progress( int act, String messageId )
            {
               fmm.logger.finer( "Beginning task:" + messageId );
            }

            @Override
            public void setMaximumValue( int max )
            {
            }
         };
         fmm.mWindowIcon =
            new ImageIcon( fmm.getResource( "images/FreeMindWindowIcon.png" ) );
         feedBack.setMaximumValue( 9 );
         ImageIcon icon = fmm.mWindowIcon;
         UIManager.put( "OptionPane.informationIcon", icon );
         UIManager.put( "OptionPane.messageIcon", icon );
         UIManager.put( "OptionPane.warningIcon", icon );
         UIManager.put( "OptionPane.errorIcon", icon );
         UIManager.put( "OptionPane.questionIcon", icon );
         fmm.init( feedBack );

         final ModeController ctrl = fmm.createModeController( args );

         // This could be improved.
         fmm.loadMaps( args, ctrl );

         Tools.waitForEventQueue();
         // focus fix after startup.
         fmm.addWindowFocusListener( new WindowFocusListener()
         {

            @Override
            public void windowGainedFocus( WindowEvent e )
            {
               NodeView selectedView = ctrl.getSelectedView();
               if ( selectedView != null )
               {
                  selectedView.requestFocus();
                  MindMapNode selected = ctrl.getSelected();
                  if ( selected != null )
                  {
                     ctrl.centerNode( selected );
                  }
               }
               fmm.removeWindowFocusListener( this );
            }

            @Override
            public void windowLostFocus( WindowEvent e )
            {
            }
         } );
         fmm.setVisible( true );
      }
   }

   public void loadTestXML( String sourceFile )
   {
      if ( test == null || !lastTestXML.equals( testXML ) )
      {
         String source = "" + sourceFile;
         System.out.println( "TestHarnessTest: "
            + TestHarnessTest.class.getClassLoader().getResource( source ) );
         InputStream inputStream =
            TestModule.class.getClassLoader().getResourceAsStream( source );
         if ( inputStream == null )
         {
            System.out.println( "InputStream not located for " + source );
            System.exit( 0 );
         }
         try
         {
            test = TestCreator.createTest( source );
         }
         catch ( Exception e )
         {
            System.out.println( "Exception " + e );
            System.exit( 1 );
         }
         lastTestXML = sourceFile;
      }
      System.out.println( "Running Test: " + test );
   }

   public void runTestScript( boolean reloadDataBase )
   {
      boolean succeeded = false;
      try
      {
         List actions = test.getActions();
         Object obj = null;
         for ( int a = 0; a < actions.size(); a++ )
         {
            Action action = actions.get( a );
            // Here we put a TestHarnessUtility with "waitFor" method, other methods.
            obj = harnessHelper.execute( action, obj ); // execute action using the selected button
            // validate if there otherwise keep looping.
            TestUtils.pause( 2l );
         }
         succeeded = true;
      }
      catch ( Exception ex )
      {
         ex.printStackTrace();
         assertTrue( "Should not fail with exception " + ex, false );
      }
      finally
      {
         loadTest = reloadDataBase;
         System.out.println( "Test: " + test + " "
            + ( succeeded ? "SUCCEEDED" : "FAILED" ) );
      }
   }

   public void runTestScriptAndNoReloadDataBase()
   {
      runTestScript( false );
   }

   public void runTestScriptAndReloadDataBase()
   {
      runTestScript( true );
   }

   @Override
   protected void setUp()
   {

      // If you need a connection, you can have it as a class variable, c, and
      // then put;
      try
      {
         harnessHelper = new HarnessHelper( fmm );
         if ( loadTest )
         {
            resetDatabase( CLEAN_DB );
            System.out.println( "Test database reset to " + CLEAN_DB );
         }
         else
         {
            System.out.println( "Test database is: " + CLEAN_DB );
         }
         Connection c = getConnection( );
         if ( c == null )
         {
            fail( "Connection not available" );
         }
         c.close( );
      }
      catch ( Exception ex )
      {
         ex.printStackTrace();
         fail( "Should not fail getting connection " + ex );
      }

   }

   protected void setUserCredentials()
   {
      String testDb = "test";
      String passwd = "tiger";
      System.out.print( "{setUp} - ");
      String dataBaseUserId = "scott";
      String url =
         "jdbc:mysql://server:3306/" + testDb;
      System.out.println( "Using database: " + url + " as user "
         + dataBaseUserId + ", resetting with: " + CLEAN_DB );
   }


   public static void main( String[] args )
   {
      TestHelper.runTests( args, ArchiverGUITestBase.class );
   }

}

In the next post, I'll show how this framework is used to actually carry out the tests. It depends upon figuring out things like: Which button on the panel is 0, 1, etc. and how do I located a menu Item in FreeMind. It doesn't set the actual display names because it is multi-lingual and does that when the menu is clicked.

At the start of the test, the init() method starts up the FreeMind application and sets up a connection to the archive database. We created the HarnessHelper referred to in the above GuiTestBase in the runTestScript( boolean val ), to actually "execute" the test methods. The loadTestXML loads in the test script and then we do the runTestScript which "executes" the actions in the test script.

So, next post will show the HarnessHelper and the results of running the test.

Posted in Code Reduction, Development, Recommendations, Software, Test Driven | Tagged , , | Comments Off

Mac OS X Setting up command line sendmail

I needed to go through a database table, pick up the email addresses and send out a reminder.
I found that while I could do that as described a bit later, the real problem was in sending the emails from
the php script. I had no way that I could send email from the command line.

I found an excellent piece of information at: Tizardsbriefcase.com

I’ll replay it here modified for my setup as it might help someone else.

 cd /usr/local/php5/lib
 diff php.ini php.ini_recommended
  ( I noticed that I needed a /tmp/crm for my mail, probably
 as a result of installing a crm product at some time ).
 sudo mkdir /tmp/crm
 edit php.ini (set the sendmail_path:)
  sendmail_path = "/usr/sbin/sendmail -t -i"
next, we need to tell postfix to set the correct permissions:
 sudo /usr/sbin/postfix set-permissions
 sudo /usr/sbin/postfix start
 sudo vim /etc/postfix/sasl_passwd
( here I endtered a gmail server using my userid on it and password )
[smtp.gmail.com]:587 myemailname@gmail.com:myPassword

 sudo chmod 600 /etc/postfix/sasl_passwd
 sudo postmap /etc/postfix/sasl_passwd
 sudo launchctl stop org.postfix.master
 sudo launchctl start org.postfix.master
 mail -s "Hello" myEmailAddress < < EOF
    Hello World
EOF

Now, the php script I used to gather the info and send out the announcement was:

< ?php
// Create connection
$dbname = 'my_database';
$link = new mysqli('my_host','db_userid','db_password', $dbname) or die('can not connect');

/* check connection */
if ($link->connect_errno) {
    printf("Connect failed: %s\n", $link->connect_error);
    exit();
}


$query = "select poc_email, name_poc from $dbname.customers ";
$result = $link->query($query);
if( !$result) {
    die('Invalid query: ' . mysql_error());
    }
if( $result !== false ) {
   printf("Select returned %d rows.\n", $result->num_rows);

while($row=$result->fetch_assoc())
    {
            $email=$row["poc_email"];
            $to = $email;
            $name = $row["name_poc"];
            $subject = "Announcement";
            $body = "Hello $name,\n\n\tHere is my\n"
                   . "\tannouncement\n";
            #mail($to, $subject, $body, $headers); // I could not get this to work!
                                                   // even if I supplied headers.
            $cmd = "echo \"$body\" | /usr/bin/mail -s \"$subject\" $to";
            shell_exec($cmd); // This worked fine with cmd above.
            echo "To=$to, name= $name\n";
    }
} else {
    echo "Error: " . mysql_error();
    die;
    }
/* free result set */
mysqli_free_result($result);

/* close connection */
mysqli_close($link);
?>

One recommendation is to put a limit 1 in the query at first and send that one to yourself to test. Make sure your formatting is right, etc.

Hope this helps. I spent quite a bit of time getting this to work and hope this will save you a bit of that time.

Posted in Installations, Problems, Recommendations | Tagged , , , , | Comments Off