apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Charles Cazabon
Hi,

I've got an odd problem that started today, and I don't see anything that is
likely to have caused it.

My setup:
  * APC SUA2200RM2U UPS, rackmount ...
  * ... connected to "master" server via USB cable, shared status via
    network to ...
  * ... "slave" server

The master server runs constantly, and has never had a problem communicating
with the UPS.  The slave server is normally powered down, as is powered up
once a week to run backups over the network, then shut down again.  The slave
has never had a problem communicating with the master, before today.

This setup has worked well since I upgraded the UPS from a smaller one a year
ago.  Nothing has changed:  still running the same apcupsd shipped by Debian
(3.14.10 with a build date of 13 September 2011).  The apcupsd config files
have not been changed in a year, the shasum of the apcupsd binary is the same
between the two servers, etc.

But today when I fired up the slave server for backups, for whatever reason,
apcupsd on the slave started sending a stream of "Warning communications lost
with UPS sua2200r" warnings and "Communications restored with UPS sua2200r"
notices, at what seems like 2-minute intervals, like clockwork:

    Broadcast Message from root@slave
            (somewhere) at 12:52 ...

    Warning communications lost with UPS sua2200r

    Broadcast Message from root@slave
            (somewhere) at 12:54 ...

    Communications restored with UPS sua2200r

    Broadcast Message from root@slave
            (somewhere) at 12:56 ...

    Warning communications lost with UPS sua2200r

    Broadcast Message from root@slave
            (somewhere) at 12:58 ...

    Communications restored with UPS sua2200r

    Broadcast Message from root@slave
            (somewhere) at 13:00 ...

    Warning communications lost with UPS sua2200r

    Broadcast Message from root@slave
            (somewhere) at 13:02 ...

The master has had no problems during the interval.

No config changes have been performed to either machine in weeks (*), and the
slave had no problems talking to the master last week.  No, there's been no
changes to the network, either, and these machines are not firewalled from
each other.  They're on the same LAN.  The machines have no problem talking
to each other over the network; the internal DNS resolves correctly for the
hostnames in question.  Nothing there has changed either.

* the slave is running a slightly newer kernel than it was last week, mainline
  4.6.7 instead of 4.6.6

While investigating this, I noticed the clock on the slave was slow and
realized I'd never installed ntp on it, so I fixed that and the clocks are now
in sync.  No change in behaviour.  I've restarted apcupsd on both the master
and the slave.  No change.

I'm a bit lost.  Nothing has changed except the minor kernel version bump on
the slave, but apcupsd this week seems to have lost the ability to maintain a
connection to the master.

Any ideas what I can do next to figure out where this is going wrong?  Am I
right in thinking the update from kernel 4.6.6 to 4.6.7 is unlikely to be the
cause?

I'll attach the apcupsd conf files for the slave and master (including some
unused defaults supplied by the Debian package), minus comments and blank
lines.  magrat is the slave; hex is the master.

I've searched the list archives, and while complaints about comms lost are
common, they all seem to be due to misconfigurations (but there's a couple
where no resolution is indicated).

Any help appreciated.  Thanks!

Charles

--
------------------------------------------------------------------
Charles Cazabon                      <[hidden email]>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------
------------------------------------------------------------------------------

_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users

master.conf (450 bytes) Download Attachment
slave.conf (495 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Ted Mittelstaedt-5
On 8/19/2016 1:20 PM, Charles Cazabon wrote:

>
> I'm a bit lost.  Nothing has changed except the minor kernel version bump on
> the slave, but apcupsd this week seems to have lost the ability to maintain a
> connection to the master.
>
> Any ideas what I can do next to figure out where this is going wrong?  Am I
> right in thinking the update from kernel 4.6.6 to 4.6.7 is unlikely to be the
> cause?


I see many people when troubleshooting try logically "proving" that some
component was not at fault.

I once logically "proved" to a co-worker that it was impossible that
a server he was troubleshooting was malfunctioning and therefore he must
be imagining things.   We had a good laugh.

Why even spend time speculating when it is so easy to just reboot the
server with the older kernel and see?


Ted

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Charles Cazabon
Ted Mittelstaedt <[hidden email]> wrote:
> >
> > Any ideas what I can do next to figure out where this is going wrong?  Am
> > I right in thinking the update from kernel 4.6.6 to 4.6.7 is unlikely to
> > be the cause?
>
> Why even spend time speculating when it is so easy to just reboot the server
> with the older kernel and see?

Because the server in question was 5 hours into a critical 20 hour job and
absolutely could not be rebooted into the older kernel.

If I can scrape the time together to try the older kernel now that that job is
complete, I will.

Charles
--
------------------------------------------------------------------
Charles Cazabon                      <[hidden email]>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Charles Cazabon
Charles Cazabon <[hidden email]> wrote:
>
> If I can scrape the time together to try the older kernel now that that job is
> complete, I will.

Ok, I brought the machine in question up, back in the kernel it was running
the previous week (4.6.6) when this definitely didn't occur.

And it's still happening:

Aug 21 15:44:45 magrat apcupsd[3060]: apcupsd 3.14.10 (13 September 2011)
  debian startup succeeded
Aug 21 15:44:45 magrat apcupsd[3060]: NIS server startup succeeded
Aug 21 15:46:16 magrat apcupsd[3060]: Communications with UPS lost.
Aug 21 15:48:56 magrat apcupsd[3060]: Communications with UPS restored.
Aug 21 15:50:11 magrat apcupsd[3060]: Communications with UPS lost.
Aug 21 15:52:51 magrat apcupsd[3060]: Communications with UPS restored.

The regular timing between the events is weird.

I haven't tried 4.7.1 yet.  That's next.

Charles
--
------------------------------------------------------------------
Charles Cazabon                      <[hidden email]>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Charles Cazabon
Charles Cazabon <[hidden email]> wrote:
>
> I haven't tried 4.7.1 yet.  That's next.

And for what it's worth, 4.7.1 behaves the same.

The two machines have no problems communicating over the network, but apcupsd
on the slave keeps losing the connection to one on the master.

Any suggestions on what to do next?

Charles
--
------------------------------------------------------------------
Charles Cazabon                      <[hidden email]>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Ted Mittelstaedt-5
On 8/21/2016 3:19 PM, Charles Cazabon wrote:

> Charles Cazabon <[hidden email]> wrote:
>>
>> I haven't tried 4.7.1 yet.  That's next.
>
> And for what it's worth, 4.7.1 behaves the same.
>
> The two machines have no problems communicating over the network, but apcupsd
> on the slave keeps losing the connection to one on the master.
>
> Any suggestions on what to do next?
>
> Charles
>

Setup a Windows system as a slave to the master and see if it also
keeps losing the connection.   If it does not then the problem is
in the slave.  If it does the problem is in the master.

If the problem is the master then you can try swapping them - that is,
reconfigure the slave to be the master, reconfigure the master to be the
slave, plug the UPS into the slave-now-master.  This would just
confirm the diagnosis.

I know the obvious thing is to blame the LAN gear but I would
assume you checked for that.

I'm going to go out on a limb and say the problem is with the master
and moreover, the problem is with the USB connection of the UPS to
the master.   But that's just a lottery guess, I would not say it's
definitive unless I was looking at the systems.  By any chance has
the USB cable on the UPS been moved to a different USB port on the server?

Ted

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Charles Cazabon
Ted Mittelstaedt <[hidden email]> wrote:
> >
> > The two machines have no problems communicating over the network, but
> > apcupsd on the slave keeps losing the connection to one on the master.
>
> Setup a Windows system as a slave to the master and see if it also
> keeps losing the connection.

There isn't a Windows system within 100 meters of this UPS - the last Windows
machine I had ran Win98.

> If the problem is the master then you can try swapping them

I might try swapping the two machines as-is.  It's a little awkward because
the machine that is the slave is only powered on one day a week.

> I know the obvious thing is to blame the LAN gear but I would
> assume you checked for that.

Yes - as I say, the two machines have no problems with any other networking.
They can talk to each other for other services, there's no packet loss, the
slave can ssh to the master, etc.  There's no network-related symptoms at all,
other than apcupsd reporting the continual loss/reestablishment of the
connection.

I'm actually about to power the slave up for this week's backups.  I'm tracing
apcupsd on the master and capturing the packets off the wire between them
today, to see if that reveals anything.

> I'm going to go out on a limb and say the problem is with the master
> and moreover, the problem is with the USB connection of the UPS to
> the master.

That's interesting, but I don't understand.  apcupsd on the master is not
reporting any problems - it isn't saying it's lost connection to the UPS or
anything - wouldn't I see diagnostics on the master if that was happening?

> By any chance has the USB cable on the UPS been moved to a different USB
> port on the server?

Nope, no changes to the physical configuration of the machine(s) in the last
year at all.

Thanks,

Charles
--
------------------------------------------------------------------
Charles Cazabon                      <[hidden email]>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Ted Mittelstaedt-5
On 8/26/2016 9:19 AM, Charles Cazabon wrote:

> Ted Mittelstaedt <[hidden email]> wrote:
>>>
>>> The two machines have no problems communicating over the network, but
>>> apcupsd on the slave keeps losing the connection to one on the master.
>>
>> Setup a Windows system as a slave to the master and see if it also
>> keeps losing the connection.
>
> There isn't a Windows system within 100 meters of this UPS - the last Windows
> machine I had ran Win98.
>

s/"a Windows system"/"a 3rd computer"/

>> If the problem is the master then you can try swapping them
>
> I might try swapping the two machines as-is.  It's a little awkward because
> the machine that is the slave is only powered on one day a week.
>
>> I know the obvious thing is to blame the LAN gear but I would
>> assume you checked for that.
>
> Yes - as I say, the two machines have no problems with any other networking.
> They can talk to each other for other services, there's no packet loss, the
> slave can ssh to the master, etc.  There's no network-related symptoms at all,
> other than apcupsd reporting the continual loss/reestablishment of the
> connection.
>
> I'm actually about to power the slave up for this week's backups.  I'm tracing
> apcupsd on the master and capturing the packets off the wire between them
> today, to see if that reveals anything.
>
>> I'm going to go out on a limb and say the problem is with the master
>> and moreover, the problem is with the USB connection of the UPS to
>> the master.
>
> That's interesting, but I don't understand.  apcupsd on the master is not
> reporting any problems - it isn't saying it's lost connection to the UPS or
> anything - wouldn't I see diagnostics on the master if that was happening?
>

There are a couple
of other issues that I think you might be missing.

First, the error message is saying communications lost with UPS, it's
not saying communications lost with apcupsd.  Second, the reality is
that your "slave" apcupsd daemon actually isn't "connected" to the
master.  It -polls- the master.   That is, it wakes up, issues a query,
gets status, then goes back to sleep until the next poll interval.  SO,
it cannot actually "lose the connection" because the connection isn't
there to lose.

Since this is Debian do the following:

netstat -atnp |grep ESTA

Do it repeatedly over and over.  Over time you will see the connections
come in and go away to apcupsd.  You won't see a persistent connection
from your slave.

Something is generating that error message somewhere.  Perhaps the slave
is not only connecting to the master it's trying to connect to a
nonexistent UPS on it's USB port?  This is why you need another system
to check the master

Ted

>> By any chance has the USB cable on the UPS been moved to a different USB
>> port on the server?
>
> Nope, no changes to the physical configuration of the machine(s) in the last
> year at all.
>
> Thanks,
>
> Charles
>


------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Charles Cazabon
Ted Mittelstaedt <[hidden email]> wrote:
>
> s/"a Windows system"/"a 3rd computer"/

Configuring another Linux machine to poll the master gives the same behaviour
as the regular slave -- same cycle of lost connection/restored connection
messages.

> > That's interesting, but I don't understand.  apcupsd on the master is not
> > reporting any problems - it isn't saying it's lost connection to the UPS or
> > anything - wouldn't I see diagnostics on the master if that was happening?
>
> First, the error message is saying communications lost with UPS, it's
> not saying communications lost with apcupsd.  Second, the reality is
> that your "slave" apcupsd daemon actually isn't "connected" to the
> master.  It -polls- the master.

I was unclear.  I didn't mean to imply a persistent TCP connection, I was just
referring to the logical connection between the slave and the master.

If I understand you, you're saying that you think the "lost
connection"/"restored connection" messages from apcupsd on the slave are
actually due to apcupsd on the master having troubles communicating with the
UPS over USB.  Is that correct?

If so, can you explain why apcupsd on the *master* is not logging any problems
of any kind?  i.e. `apcaccess status` on the master gives me a current-looking
status report as normal, and the apcupsd.events log just shows "the usual":

[...]
2016-08-04 17:42:09 -0600  UPS Self Test completed: No test results available
2016-08-09 10:06:44 -0600  Power failure.
2016-08-09 10:06:49 -0600  Power is back. UPS running on mains.
2016-08-19 10:13:35 -0600  apcupsd exiting, signal 15
2016-08-19 10:20:35 -0600  apcupsd 3.14.10 (13 September 2011) debian startup
succeeded

The 19th is when I tried restarting apcupsd on the master.

i.e. I took the above (no errors logged on the master, normal status output
from apcaccess) to mean the master was successfully communicating with the
UPS.  Am I wrong about that?

Thanks,

Charles
--
------------------------------------------------------------------
Charles Cazabon                      <[hidden email]>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Ted Mittelstaedt-5
On 8/26/2016 2:47 PM, Charles Cazabon wrote:
> Ted Mittelstaedt <[hidden email]> wrote:
>>
>> s/"a Windows system"/"a 3rd computer"/
>
> Configuring another Linux machine to poll the master gives the same behaviour
> as the regular slave -- same cycle of lost connection/restored connection
> messages.
>

Then the problem is in the master.

>>> That's interesting, but I don't understand.  apcupsd on the master is not
>>> reporting any problems - it isn't saying it's lost connection to the UPS or
>>> anything - wouldn't I see diagnostics on the master if that was happening?
>>
>> First, the error message is saying communications lost with UPS, it's
>> not saying communications lost with apcupsd.  Second, the reality is
>> that your "slave" apcupsd daemon actually isn't "connected" to the
>> master.  It -polls- the master.
>
> I was unclear.  I didn't mean to imply a persistent TCP connection, I was just
> referring to the logical connection between the slave and the master.
>
> If I understand you, you're saying that you think the "lost
> connection"/"restored connection" messages from apcupsd on the slave are
> actually due to apcupsd on the master having troubles communicating with the
> UPS over USB.  Is that correct?
>

Yes.

> If so, can you explain why apcupsd on the *master* is not logging any problems
> of any kind?

Because the problem is a bug in apcupsd?  Or because it's something
external to apcupsd.

For example your master server could be a guest OS on a hypervisor.
It could be completely suspended, and swapped out to disk, for a long
period - minutes perhaps - then swapped back in and executing.  It would
have no knowledge of a problem - but anything attempting to poll it
during that time would have gotten a no-response.

I've seen systems with hardware issues that periodically would just
freeze - the entire system would just halt - for a second or so then
pick right up without even realizing it had frozen.

You now know the problem is on the master.  You know that it just
started happening recently - but you have not had any software
changes that would have been responsible - or you have back-reved
to old versions and the problem still exists.  SO you have eliminated
software so you are left with hardware.  And that hardware must be
hardware on the master.

  i.e. `apcaccess status` on the master gives me a current-looking
> status report as normal, and the apcupsd.events log just shows "the usual":
>

If your system was swapped to disk on a hypervisor then there would be
no errors on an apcaccess status because apcaccess would be swapped at
the same time your apcupsd was not responsive.

If your system picked up a rootkit that put itself in between the
actual hardware and your Linux system the same thing would happen.

If you inserted a hardware card with a driver problem or a USB stick
with a driver problem then the driver could be triggering your hardware
to halt for a second then get going again due to some sort of bug -
same issue.

If your hard disk suddenly got marginal sectors and when your OS made a
sector call to the disk and the disk spent 30 second scrubbing on a
sector before returning data, this might cause this and your OS might
not even be aware of a problem because the disk lied to it.

if you have a scratch Linux system - put apcupsd on it, plug in your
UPS and see if it works.

Ted

> [...]
> 2016-08-04 17:42:09 -0600  UPS Self Test completed: No test results available
> 2016-08-09 10:06:44 -0600  Power failure.
> 2016-08-09 10:06:49 -0600  Power is back. UPS running on mains.
> 2016-08-19 10:13:35 -0600  apcupsd exiting, signal 15
> 2016-08-19 10:20:35 -0600  apcupsd 3.14.10 (13 September 2011) debian startup
> succeeded
>
> The 19th is when I tried restarting apcupsd on the master.
>
> i.e. I took the above (no errors logged on the master, normal status output
> from apcaccess) to mean the master was successfully communicating with the
> UPS.  Am I wrong about that?
>
> Thanks,
>
> Charles
>


------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: Solved: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Charles Cazabon
Ted Mittelstaedt <[hidden email]> wrote:
> > If I understand you, you're saying that you think the "lost
> > connection"/"restored connection" messages from apcupsd on the slave are
> > actually due to apcupsd on the master having troubles communicating with the
> > UPS over USB.  Is that correct?
[...]
> You now know the problem is on the master.  You know that it just
> started happening recently - but you have not had any software
> changes that would have been responsible - or you have back-reved
> to old versions and the problem still exists.  SO you have eliminated
> software so you are left with hardware.  And that hardware must be
> hardware on the master.

I couldn't find any hardware problems with the machine in question, so when I
powered up the slave today, I started watching what was going on with both
machines.  And lo and behold, apcupsd was switching from normal sleep state
(S) to uninterruptible sleep state (D) periodically, and those switches were
when the slave was reporting loss of connection to the UPS.  After a couple of
minutes, apcupsd would switch back to normal sleep, and the slave would report
the connection restored.

It appears that the USB subsystem/driver must have gotten wedged somehow,
causing apcupsd to go into uninterruptible sleep waiting for a response from
the kernel.  Not sure why, but it looks like a Linux kernel/driver problem:
software, but not anything to do with apcupsd.

Force-resetting the usb controller (aka unplug/replug ;) restored it to normal
operation.

Thanks for your assistance, and your patience.

Charles
--
------------------------------------------------------------------
Charles Cazabon                      <[hidden email]>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: Solved: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Ted Mittelstaedt-5
On 9/2/2016 4:11 PM, Charles Cazabon wrote:
>
> Force-resetting the usb controller (aka unplug/replug ;) restored it to normal
> operation.
>

OK so then just to confirm - on your system, if you do a cold-boot
then it will come up messed up - and after it's booted, you can just
unplug the USB cable and plug it back in and it will start working
normally?   And, you have to do this on every cold boot?  Or did just
doing this once fix it for future cold boots?

I think we had someone report something similar a year or two ago.

> Thanks for your assistance, and your patience.
>

You are quite welcome!  I had a feeling USB was at the bottom of it.

We want to get this USB stuff as documented as we can because the USB
standard itself is loose and the implementations are extremely messy.
It is as bad as the old hardware bugs we used to see in different
versions of serial port chips.   But because so many UPSes are now
using USB connectivity this problem is going to continue in the future.

Ted

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users
Reply | Threaded
Open this post in threaded view
|

Re: Solved: apcupsd 3.14.10 (Debian) suddenly started losing/restoring connection to master

Charles Cazabon
Ted Mittelstaedt <[hidden email]> wrote:
> >
> > Force-resetting the usb controller (aka unplug/replug ;) restored it to
> > normal operation.
>
> OK so then just to confirm - on your system, if you do a cold-boot then it
> will come up messed up - and after it's booted, you can just unplug the USB
> cable and plug it back in and it will start working normally?

Er, no.  The UPS is connected to the master via USB.  The master is a server
which is never powered down.  It's only rebooted to start running a newer
kernel, which I do two or three times a year, but that's a warm boot.  The
slave server is the one which is powereed up and down regularly, but it just
talks to the master over the network.

I've never seen problems with USB on this machine before (in five years or
more), but the only thing connected to the server via USB is the UPS, so
there's not much else to go wrong.

I presume USB would would work normally at a cold boot, just like it normally
does.

> I had a feeling USB was at the bottom of it.

Well, you were definitely right.

> We want to get this USB stuff as documented as we can because the USB
> standard itself is loose and the implementations are extremely messy.

Indeed, it is.  I don't know if this will help, but the USB controller(s) on
this machine come from the Intel 5000P chipset, identified by lshw as:

    description: USB controller
    product: 631xESB/632xESB/3100 Chipset UHCI USB Controller #1
    vendor: Intel Corporation
    version: 09
    configuration: driver=uhci_hcd latency=0

Charles
--
------------------------------------------------------------------
Charles Cazabon                      <[hidden email]>
Software, consulting, and services available at http://pyropus.ca/
------------------------------------------------------------------

------------------------------------------------------------------------------
_______________________________________________
Apcupsd-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/apcupsd-users