Hi All,
Just recently updated to:
ICG 2.01
UMS 6.02.110
ICG loses connection.
This has been the 2nd time this has happened since the upgrade(2 days ago).When I check the ICG in UMS administration it takes >2min to show me the status (the freeze usually indicates the ICG is not connected)
The work around is to reboot the UMS server and ICG Appliance. This is not a stable work around to do every morning. I expect this to happen again tomorrow morning.
Here I would highly recommend to open a ticket, since it might get useful to get 3rd level involved. I will try to check if I have some ideas since I believe you already checked everything like Firewall, Antivirus, etc, right?
I just sent the Catalina logs to L3 support. Weird that it doesnt happen throughout the day. Just early mornings.
Sounds weird, really… Are they maybe some Backup/AV scan jobs running at this moment (on UMS, ICG, DB)?
How many devices are you using? Is it possible that a huge amount (1k<) of devices are starting at one moment?
We’re having the exact same issue after upgrading to 2.01. Looks like it stopped taking requests at about 2-3am this morning and went into disconnected state. A simple reboot fixes it for about a week.
One of our engineers opened a ticket for this – I will follow-up on status.
@member Has L3 support gotten back to you on what the issue could be?
yes, an engineer reached out and the dev team provided a conf file for our ICG. I was getting the same behavior in the early AMs. With the new conf file in place we’ll see tomorrow if it is stable.
@member sorry for the late reply, we have about 3,500 devices coming in through the ICG
+ @member for visibility
Yeah as you can see we aren’t near that scale and have similar issues. Hopefully the config file helps.
@member Can‘t wait to get your feedback🙏
hi @member @member Looks like the new conf file did the job. I will need Jason to take a look at the Catalina logs as I saw some errors.. But logged on first thing in the morning and its connected. Can you tell from the screenshot that it was down intermittently in the early AMs again? due to all the requests or is that normal?
ICG is down again, now during the day. Around 3:40pm PST today. With the new Conf file installed yesterday
now with rebooting the ICG and UMS server, it is back to normal…
We only have to reboot ICG; don’t have to reboot UMS server.
@member I think that doesnt work for us, i will try that if it goes down today again..
Cool – we have a call with Jason this afternoon as well. Hope to get quick resolution to this.
@member ICG was down again, early AM and just rebooting the ICG was fine. Back up now. Are you issues still persisting?
Yes – we did not get the config file since it didn’t not appear to fix your issue. Ours happens every few days versus every day, but it may be because you have many more devices hitting your ICG than ours.
Definitely…. I hope it gets resolves soon.
Has the new config file made any difference for you? We increased the heapsize up to 4GB as well as increased the VM memory to 4GB.
Made it somewhat stable… We found that our disk space was at 100% capped at 20GB. The .lobs file took up 18GB. That .lobs file fills up from rebooting devices/checking in and deploying profiles. I guess 3,500 devices through ICG could really kill the disk space. We’re building a new ICG server with 100GB and latest Ubuntu.
Thats not the root cause for the ICG to go down once in a while but its something that we’re fixing at the moment and hopefully solves for the disconnect
Hi Guys, since a few minutes they are two new versions available UMS 6.03.100 and ICG 2.01.110. Both include enhancements regarding memory management, it’s worth a try.
Yep just saw this that morning – the config file change actually seems to have fixed it for us for 2.01.100!
Hey @member and @member I actually just submitted a ticket for the same issue. Do you guys happen to have the config file and where to place it? Thanks again in advance.
@member, Could you assist? I have an existing ticket but still waiting for someone to call. Need to do some presales for a potential client who are on ICG but having this issue.
@member Will try to! Which kind of issues are you facing?
Which UMS / ICG Version are you using?
Latest on both. ICG 2.01.100 and UMS 6.03. Just updated ICG last week. Started noticing the issue right away but didn’t think much of it until today.
Also having trouble sending commands/profile and configurations to devices that’s connected to ICG. Seems only reboot command works.
And how much devices are affected roughly?
Which Igel OS Version?
I say about 18’ish. 11.02 and 10.05
Hem… 18 isn‘t covering the issue described above (we spoke mostly about >200 when having heap size issues).
Is there any kind of antivirus / Firewall in between?
Correct, they’re a larger client so I am assuming that’s not the case as well. We have Palo Alto Traps on the UMS server but that’s it. It was working fine from what I can tell until the ICG upgrade to 2.01.100.
Hi @member,
I have 2 ICG servers.
ICG2 has about 3,800 devices connected and disconnects once a day, twice the most.
ICG3 has about <100 devices connected and no disconnects.
I have a meeting with IGEL on upgrading to ICG-2.01.120 and UMS 6.03.110 and see if ICG2 is stable..
We also have Palo Alto FWs as well.
@member is 2.01.100 holding up for you? any disconnect lately? and how many devices are connected now?
Do you happen to have the config file they sent by any chance and I can test it out? I also requested 2.01.110 to see if that does anything.
I would consult with your presales engineer and probably go to 2.01.120
the conf file that was tested did not work and it was back in Sept. there has been 2 more ICG roll outs since then
Yeah, I will try the .120 version and see how it goes.
Thanks!
Just for info: igelcommunity.slack.com/archives/C8FC0D6U9/p1575931164107100
Thanks Sebastien. I have a different issue on top of the old one but I spoke to you about it already. Let me know where they are at with my support ticket. Thanks again!
Hi All,
@member @member
Yesterday we upgraded our ICG server to 2.01.120 and added 150GB to the /opt folder (since we have 3,800+ on that ICG)
Came in the office this morning expecting it to disconnect as usual but it is still up.
We will monitor the iCG this week then we will also upgrade our UMS to 6.01.130
But I suggest to upgrade your ICG to 2.01.120 and see if that resolves your issues
Thanks for the info and suggestion! I’ve requested access to the private build, and am awaiting this access. In the meantime, I rebuilt the ICG (again, I am on AWS free-tier Ubuntu) and installed ICG 2.01.100. Everything is stable again, it seems. As a bonus, this build I used Amazon Linux (CentOS-based) and it worked like a charm.
_(Another thing to note: The memory leak in 2.01.110 in the notes state only Debain may be affected. I may try to build .110 on Amazon Linux and see what happens)_
How long did it take you to rebuilt as I have never done it before. Another engineer did for us. @member
JFYI, 2.01.130 just released, might look into that this week when I have my check in with IGEL. So far so good on .120 though
Great question! Actually, it was less than 10min. A big piece of this was because I was running in the cloud, which enabled quick turnaround on providing a stable Linux OS:
1 – Picked an OS to substantiate (via Amazon’s Marketplace of templates) – called an AMI
2 – Walked through the AMI wizard in Amazon to set this up (this included firewall rules to allow for ports 22 (SSH) and 8443 (IGEL ICG/UMS access) to the instance
3 – Once running, I SSH’d in, and ran all updates (apt-get update/yum)
4 – After, I ran through this IGEL KB to ensure Python was running and installed:
<kb.igel.com/igelicg-2.01/preparing-the-linux-machine-19181657.html>
python -V
5 – Then SCP’d over the installer.bin and Keystore file to the /tmp
6 – Modified the .bin to execute, and ran the installer:
sudo chmod u+x installer-[version].bin
./installer-[version].bin keystore.icg
7 – Walked through the ICG setup wizard
8 – Went back to my UMS, removed any current ICG instance (clicking the heading IGEL Cloud Gateway)
9 – Reconnected my new ICG by adding it in to the UMS and pointing it back to my cloud ICG via the UMS ICG add wizard
A few things to note:
• In the cloud, allocate a static IP/DNS mapping to your ICG OS instance (AWS-speak called an elastic IP)
• Ensure your security groups are setup correctly (AWS-speak called security groups that provide things like firewall access/etc to certain VPCs)
• Ensure you have an identity access management key pair group setup for this. It’ll make SSHing to the instance seamless
Awesome! I might give this a shot. Thanks @member!!!
Sorry @member, I wasn’t keeping track of this thread. I believe the newest ICG has the new config file with the build. It just adjusted the max memory size for the app.
Gotcha. We have the newest ICG and still seeing the issue where devices are dropping unless we reboot it.
Continue reading and comment on the thread ‘IGEL OS lossing connection with ICG’. Not a member? Join Here!
Learn more, search the IGEL Knowledge Base
Ask a question or comment on the above message thread?
Join or log in to the IGEL Community to ask us anything and meet other IGEL customers, partners, and EUC enthusiasts.Submit a question, or Join Today!