*Resolved* Add Service down?

[joarb]

I'm getting a HTTP 500 when I click the "Add Service" button after I've filled out the form in https://dashboard.moaicloud.com/createapplication.php. Would be thrilled if someone could give that server the love and tenderness it deserves.

[joarb]

Problem seems to be solved now.

[joarb]

I've been trying to edit my newly created service. Most of the time, I've gotten HTTP 500's when clicking on the "Edit files" in the "Resources" category of the dashboard. When I do get to edit the files, I'm not able to save my edits. The error message is as follows:

Warning: MongoGridFS::findOne() expects parameter 2 to be array, boolean given in /var/ahu/deploy/releases/894727897921ed90db7c65fb201dc71011a1fc78/www/classes/MongoUtils.php on line 92

I'd be really grateful if someone could have a look at this.

[joarb]

I'm happy to report that these problem seems to have been resolved.

[todd]

Thanks Joarb. Rob is looking at it...

[rob]

It looks like someone was up late sat. night.

There are two issues from this event that we've made work items for. First a quick recap of what occurred, at 1:57 am PST our primary and secondary databases swapped roles. They then swapped back at 6:08 am. This shouldn't be anything that is customer visible since the cluster is designed to handle changes in the primary. However it seems like the php driver, or our usage of it does not handle that situation gracefully.

I've made a small change to our usage of the driver that might resolve this issue. I'll test it with a manual failover of our testing cluster on monday. We've also had a ticket open with our database vendor for a similar issue. Additionally whenever the dashboard encounters an error, it should notify us while presenting the user with a 500 page. We did receive some notifications, but it seems the 500 page was not functioning as expected since users should never receive raw error messages. That is a second issue that we'll look to correct. Sorry for any interruptions.

[joarb]

Thanks for your detailed response, rob!

Now, I'm sorry to say that I've ran into other issues as well. I'm getting quite a lot of unintelligible error reports like this:

Stack Trace
/usr/local/share/lua/5.1/tir/mongrel2/connection.lua:48:
bad argument #1 to 'length' (string expected, got nil)
Source Code
not available

I haven't yet found a pattern of what causes it, but when I do I'll let you know. In the mean time, I'd be extremely grateful if you guys could have a look at this.

[joarb]

Well, this was kind of embarrassing. It complained, and rightly so, about this:

Code: Select all

1. local params = web.params ()


2. if ( params.someParam = "someValue") then


3. ...

where it should've been a "==" operator in the conditional. Would've been nice with a slightly more informative error message, though.

[anddrewscott]

Well, this was kind of embarrassing. It complained, and rightly so, about this:

Code: Select all

1. local params = web.params ()


2. if ( params.someParam = "someValue") then


3. ...

where it should've been a "==" operator in the conditional. Would've been nice with a slightly more informative error message, though.

That one always gets me.
(Related:)
It took me a while how to work out how to do:

Boolean = someVar = sameValueVar

of course, the second equals has to be a double one.

[xseb360]

Seems to be down again ?

I get HTTP 500 when trying to Add Service (creating the Hello Tutorial).

[Ezra]

It should be up and working. Does your 500 page say anything? Our error logs aren't showing a thing.

[xseb360]

Server error
The website encountered an error while retrieving https://dashboard.moaicloud.com/createapplication.php. It may be down for maintenance or configured incorrectly.

HTTP Error 500 (Internal Server Error): An unexpected condition was encountered while the server was attempting to fulfill the request.

I get this after clicking Add Service here:

[todd]

Rob is out today but we are reviewing this. Stay tuned.

Thanks

Todd

[xseb360]

Thanks, let me know if you need more info.

[rob]

Thanks for reporting this. Our system is set up to email the dev team whenever the cloud reports a 500 error. We don't get many, but lately we've been seeing more than I would like (1 is more than I would like). One issue was caused by the attempted fix to the first problem reported in this thread. That change has been rolled back. Your issue is a different one, and I'd appreciate it if you could dm the email address you signed up with so I can look into it a little more.

[rob]

@xseb360 I found the emailed error report(s) from your session. Your issue is actually the same as the one mentioned earlier in this thread. Basically there was a race condition within our dashboard between writing data to our primary server and then reading it back from our secondary. If the data did not replicate across then they system wasn't able to generate an id for you. That lack of an id later causes an issue when you try to create a service. The root cause of this issue was removed from the dashboard, I have a little bit of data cleanup to do now. It looks like there are 33 affected users and you. Sorry about that.

[xseb360]

It's working now.

Thanks!