Tuesday, August 16. 2016
Integrating runalyze into runnerup (Part II)
This is the second part of the series about the integration of runalyze into the android application runnerup. As you know runnerup is an opensource application to track your running activities in any android phone. The runalyze project is the other side of the coin, it is an opensource (license is not already specified but the project claims it will be AGPL) server to store and analyze your exercises. So, for me, as an amateur runner, the integration between the two projects is a must.
There is an open issue in runnerup to perform the integration that sadly I missed for a very long time. There people from runalyze explains that they are working in a next version 3.0 and then the idea is starting a restful web services API for integrating external applications. The issue recommends to wait for that API and then perform the integration between the two projects. On the other hand, I started another project that was my little runalyze called runnerupweb. I did that just because I had not find any software of that type (as I said I did not know anything about runalyze). As I stated before I think that the integration between both applications is very important and, therefore, I started to study how it could be (see the previous entry in the series). Finally I decided to not wait for the API and implement something that just let you upload your activities in an easy way.
And it is almost done! The past weekend I finally had the time to implement the proposed integration presented in the previous entry. It works, there are some tricky parts (how the IDs are converted from one application to the other, calculating the calories of the activity, and so on) but the synchronizer is able to upload your tracks from the phone to the server. Please take in mind that my main target is using my own runalyze server, so the solution MUST deal with that feature. (As it was commented when I was implementing my runnerupweb application, the phone application just manages standard cloud services and therefore the different accounts just request username and password. In order to have your own backend it is absolutely necessary to also request an URL. In this case I have just done exactly the same idea I did in the case of runnerupweb, a new authentication method was created that requests url, username and password. The only difference in the new integration is that now the URL is filled by default with the public URL of the runalyze services. You have to be kind with people that work for the community.)
Finally I commented in the github issue that a working solution is in place. The user mipapo (a runalyze developer) told that the solution would only work with current version 2.x, and not for the upcoming version 3.x. It seems that the new version changes some internal framework or something and much of the URIs are different from one version to the other (you know, the URL for login or the URL for uploading the activity). So, I had to spend two hours more to change the synchronizer to detect the version, perform the login and upload the activity to the proper end-point. Now the implementation works with both versions, and you can smoothly upgrade runalyze from 2.x to 3.x without worrying about runnerup, the synchronization will work without changing anything.
Just one more comment, the runnerup application uses a smart technique to perform the login and later calls. It stores the cookies after a login and resend them in the next operations (just like a browser does). I had faced a problem in the implementation. The runalyze version 2.x returns two PHPSESSID cookies (the cookie that handles the session and in turn maintains the login) if you access the login.php directly to submit the username and password. I think this is a bug that should be fixed by runalyze but for the moment I overrided the getCookies method in the synchronizer to get the last cookie (that way it works).
This is a little video uploading some activities with the development environment into a local 2.6 runalyze server.
And that is all. Now I am waiting what is the reaction in the runnerup project. If you are interested please try to read the issue and test the application I uploaded to my github (remember that this is at your own risk and do not forget to do an export before anything else). I will try to handle any inconvenience with the synchronizer (but be merciful, I have little experience in android and even less time).
That's the way I wanna open source!
Friday, June 24. 2016
Integrating runalyze into runnerup (Part I)
As Marx2 told me in the github pull request commented in the previous entry, the runalyze project is really nice. The previous weekend I was testing and uploading some of my running activities just to try it. And, in general, the impression is quite good. Obviously is much more complete than my little application RunnerUpWeb and works smoothly (actually the application is much more than what I needed and looked for).
At the same time I studied how the files were added (just in case I have the time to integrate it inside runnerup phone application). I tested with TCX files (the ones I usually use in my application and, therefore, I had easy access to them) and the process that runalyze uses is, at least, strange. When you want to upload any type of file the procedure that is followed is more or less the following:
When the upload button is clicked you can choose a file from your file system (a lot of formats are allowed, TCX among them).
The file is uploaded using a multipart/form-data post but nothing more, I mean, the file is uploaded to a directory inside the server but no real parsing is done. The file is just placed in the system (in a folder of the system more concretely).
After the file is uploaded another request to the server triggers the parsing (TCX parsing in my case). And the result of this operation is creating a huge HTML form with a lot of inputs (the real data obtained from the TCX file but exposed as a form). The HTML form is incrusted in the page as another tab in the import screen.
In that form the user can modify some inputs of the activity (some information that can be added or modified to the parsed data).
Finally the user submits that form and the activity is actually saved in the system (inside the ddbb).
At first I was thinking about replicating the same procedure with runnerup but it is a waste of time. I think now that the most proper way of integrating runnalize into a runnerup is just constructing the post data directly (all the needed input that the import process generates from the TCX file in the web application) and sending that to the runalyze. This idea means that the phone should generate a proper form data to post and only executes the last point of the procedure (point 5 in the previous explanation). Obviously that information should be generated reading the activity data stored in the phone database (exactly as it is done for generating a TCX or GPX file).
I am going to add here the java method that was tested to send the data (raw data copied from the browser) to the runalyze server.
public void submit() throws Exception {
URL url = new URL(url + "/call/call.Training.create.php?json=true");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setInstanceFollowRedirects(false);
conn.setDoOutput(true);
addHeadders(conn); // set the session cookie into the request
try (OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream())) {
StringBuilder sb = new StringBuilder()
.append("creator=").append("&")
.append("creator_details=").append("&")
.append("activity_id=1461954855").append("&")
.append("timezone_offset=120").append("&")
.append("arr_time=2|4|6|...").append("&")
.append("arr_lat=40.463737|40.463726|40.46372|...").append("&")
.append("arr_lon=-3.7070801|-3.7070918|-3.707106|...").append("&")
.append("arr_geohashes=").append("&")
.append("arr_alt=754|753|753|...").append("&")
.append("arr_alt_original=").append("&")
.append("arr_heart=78|80|80|...").append("&")
.append("arr_dist=0|0.00161|0.00309|...").append("&")
.append("arr_cadence=").append("&")
.append("arr_power=").append("&")
.append("arr_temperature=").append("&")
.append("arr_groundcontact=").append("&")
.append("arr_vertical_oscillation=").append("&")
.append("arr_groundcontact_balance=").append("&")
.append("pauses=[{\"time\":302,\"duration\":2,\"hr-startv\":156,\"hr-end\":156},{\"time\":570,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":848,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1138,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1306,\"duration\":4,\"hr-start\":0,\"hr-end\":0},{\"time\":1448,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1722,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1908,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1912,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":2038,\"duration\":2,\"hr-start\":0,\"hr-end\":0}]").append("&")
.append("hrv=").append("&")
.append("fit_vdot_estimate=0.00").append("&")
.append("fit_recovery_time=0").append("&")
.append("fit_hrv_analysis=0").append("&")
.append("fit_training_effect=").append("&")
.append("fit_performance_condition=").append("&")
.append("elapsed_time=2288").append("&")
.append("elevation_calculated=0").append("&")
.append("groundcontact=0").append("&")
.append("vertical_oscillation=0").append("&")
.append("groundcontact_balance=0").append("&")
.append("vertical_ratio=0").append("&")
.append("stroke=").append("&")
.append("stroketype=").append("&")
.append("total_strokes=0").append("&")
.append("swolf=0").append("&")
.append("pool_length=0").append("&")
.append("weather_source=").append("&")
.append("is_night=0").append("&")
.append("distance-to-km-factor=1").append("&")
.append("sportid=1").append("&")
.append("typeid=1").append("&")
.append("is_race_sent=true").append("&")
.append("time_day=29.04.2016").append("&")
.append("time_daytime=18:34").append("&")
.append("s=37:46").append("&")
.append("kcal=554").append("&")
.append("pulse_avg=152").append("&")
.append("pulse_max=174").append("&")
.append("distance=7.851").append("&")
.append("elevation=0").append("&")
.append("pace=4:49").append("&")
.append("power=0").append("&")
.append("cadence=0").append("&")
.append("splits[km][]=1.006").append("&")
.append("splits[time][]=5:00").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.005").append("&")
.append("splits[time][]=4:30").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.004").append("&")
.append("splits[time][]=4:40").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.000").append("&")
.append("splits[time][]=4:52").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.004").append("&")
.append("splits[time][]=5:16").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.003").append("&")
.append("splits[time][]=4:36").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.004").append("&")
.append("splits[time][]=5:22").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=0.875").append("&")
.append("splits[time][]=3:50").append("&")
.append("splits[active][]=1").append("&")
.append("use_vdot=on").append("&")
.append("rpe=").append("&")
.append("comment=").append("&")
.append("partner=").append("&")
.append("route=").append("&")
.append("notes=").append("&")
.append("weatherid=1").append("&")
//.append("temperature=").append("&")
.append("wind_speed=").append("&")
.append("wind_deg=").append("&")
.append("humidity=").append("&")
.append("pressure=").append("&")
.append("tag_old=").append("&")
.append("equipment_old=").append("&")
.append("submit=submit");
writer.write(sb.toString());
writer.flush();
conn.connect();
String response = getResponse(conn.getInputStream());
if (conn.getResponseCode() != 200 || !response.contains("The activity has been successfully created.")) {
throw new IllegalStateException("Error code " + conn.getResponseCode());
}
}
}
The java class uses a technique to save the headers of a previous login (as runnerup is doing right now) and add them to the next submit. That login method is not shown in the entry. As you see the activity data is sent using a lot of inputs. Some of them are arrays of numbers separated by a pipe (all the data for samples: time, latitude, longitude, altitude, distance,...). Others are just some fixed data (as the day and time of the activity, the pace, the sport and type,...). For example the information about pauses is more complicated (it is a json with time, duration and heart-rate at the beginning and the end of each pause). One input line is commented out (temperature), just to check if you can omit optional fields. If you sent all that information a new activity is created in runalyze. So now the problem is obtaining all those inputs from the internal runnerup database for the activity to export. My main concern about the process is how much stable it will be. Will runalyze change the way activities are created? Is this form more or less stable? If someone involved with the project reads the entry please comment below or contact me directly.
In general now I am quite busy and I do not know if I will have the time to develop all the synchronizer stuff. I am just creating this entry to not forget what I have already thought and to encourage any of you to also start the task. The problem seems to be just reading from the database and filling the post data (but you know that it will never be so easy at the end). I will try to develop the synchronizer but I cannot say when I will do it. So, if you have the time, please go for it.
Regards!
Saturday, June 11. 2016
Making the changes for RunnerUpWeb
If you think that the RunnerUpWeb project was forgotten or stopped you could not be more wrong. I have been busy the previous weeks trying to integrate the backend as a regular account in the android application. As a result now there are two pull requests waiting in the github repository:
A change to make that the runnerup application generates valid TCX files (compatible with the associated XSD). This way the application ensures that the file, uploaded via phone or browser, is valid and not another (maybe dangerous) thing.
The last pull request adds a new account that represents your own RunnerUpWeb installation. This is the main enhancement to make the phone app understand a RunnerUpWeb backend.
Obviously the second point is the important one. Integrating the account into the phone counterpart is the main step to make this project start up (anyone would be able to configure easily his RunnerUpWeb account in the phone). Once this point was overcome I can continue developing the backend application in order to release a first (more or less) stable version.
Adding the account was much more difficult than expected because normal backends in runnerup have a fixed URL to connect to (Digifit, endomondo, sports-tracker and so on always connect to the same hostname for storing the activities). Nevertheless RunnerUpWeb is a FOSS application you can install wherever you like (your home PC, your hosting web server,...), so the URL the app should connect to is variable and not fixed like in all the other accounts. For the moment I have implemented an easy workaround. Until now each account can store an authentication configuration (username and password), with my request there is another type of authentication that adds the URL of the service to the previous information (username, password and URL are now saved in the auth config). With that change I could extend the former DigifitSynchronizer class with some minor changes to make it compatible with my application.
A second nasty issue (but this one I think it is caused because my complete ignorance programming in android) was that the image for my account were reassigned to other accounts. Finally I understood that the application was assigning the images in (lexicographic) order and my a16 image was before a2, a3 and so on. I renamed my logo to b01_runnerupweb.png and I think everything works as expected now. But I suppose that people from runnerup project will recommend me something better for this problem.
I present a little video using the android studio to configure and connect a virtual device to my development environment. First the modified application is deployed to the phone. Then I show that the new account is there ready to be configured. A new user is created in the backend server. The account is created against the demo server and some activities are uploaded. Finally the user logs in the backend server and the activities are there. Really nice, isn't it?
Some hours after my pull request was posted, Marx2 commented about Runalize. It seems that it is another try to make another free source backend server for sport activities. I simply did not find it when I looked for that type of application. Let's see what happens because Runalize has no account in runnerup neither. My personal opinion is that they should have been trying what I am doing long time ago. That way they have saved me a lot of time and effort. For the moment I will keep pushing to integrate my changes in runnerup. I suppose that Runalize also needs URL configuration for the account. So the change can help both.
Enjoy!
Comments